EKS + EBS Storage with eksctl

Provision EKS cluster + EBS CSI using eksctl

Earlier I wrote an article how to standup a EKS cluster with a few commands using eksctl. I also demonstrated how to add a AWS EBS CSI driver, so that you can use persistent volumes. This is important because this feature for storage no longer comes standard starting with EKS 1.23 and beyond.

In this article, I show you how to do this properly, more appropriate for production for the scope of persistent volume feature. This article shows how to install this using either:

• Helm chart
• EKS add-on feature

Background on securing cloud access

As most know by now, Kubernetes is a platform that can schedule (launch) applications that are composed as one or more containers packaged up as a pod.

What may not be obvious, is that Kubernetes is an abstraction layer to cloud resources, where Kubernetes will provision cloud resources like storage and networking (load balancers, reverse proxies). So essentially, Kubernetes is similar to Terraform in the scope that both can provision cloud resources. For this to work, Kubernetes needs privileges to access the cloud resources.

Initially (before 2019), access to cloud resources was typically granted to all containers (pods) running on Kubernetes cluster. This obviously can be quite dangerous, inviting potential abuse as access is granted to all applications whether or not such access is needed. Ideally, access should only be granted to the pod that needs such access, a concept called PoLP (principle of least privilege).

Fortunately, this concept is possible using the facility officially called AWS Identity and Access Management (IAM) roles for service accounts or IRSA for short. In this process, you set up a OIDC provider that is used to establish a trust relationship between identities: a Service Account on Kubernetes and an IAM Role on AWS.

In AWS, an IAM Role is an identity that used in automation. You will create a policy that contains the permissions, scope, and conditions to apply the permissions and attach this policy to the IAM Role. In Kubernetes, an analogous identity would be the Service Account, which is assigned to the desired application (pod).

Once these two identities are created and the AWS IAM Role and the Kubernetes Service Account are associated with each other, then only the pods that are assigned to the Service Account will have access to the targeted cloud resources.

With AWS EBS CSI (container storage interface) driver, only the ebs-csi pods will need access to the EBS API will be granted access through the IRSA facility.

Related Articles

Previous Guide

This tutorial is minimalist set of commands to set up an EKS cluster.

Kubernetes (EKS) with eksctl

The asdf tool

This tool is used to manage different kubectl versions. It is a easy way to switch between kubectl versions. This article includes details on how to run earlier amd64 binaries on newer M2 Macbooks that use arm64.

Managing the many kubectl versions

Prerequisites

Tools

These are the tools used in this article.

• AWS CLI [aws] is a tool that interacts with AWS.
• kubectl client [kubectl] a the tool that can interact with the Kubernetes cluster. This can be installed using adsf tool.
• helm [helm] is a tool that can install Kubernetes applications that are packaged as helm charts.
• eksctl [eksctl] is the tool that can provision EKS cluster as well as supporting VPC network infrastructure.
• adsf [adsf] is a tool that installs versions of popular tools like kubectl.

Additionally, these commands were tested in a POSIX Shell, such as bash or zsh.

GNU Grep was also used to extract version of Kubernetes used on the server. On Linux will likely have this installed by default, while macOS users can use Homebrew to install it, run brew info grep for more information.

AWS Setup

Before getting started on EKS, you will need to set up billing to an AWS account (there’s a free tier), and then configure a profile that has provides to an IAM User identity. See Setting up the AWS CLI for more information on configuring a profile.

After setup, you can test it with the following:

export AWS_PROFILE="<your-profile-goes-here>"
aws sts get-caller-identity

Kubernetes Client Setup

If you use asdf to install kubectl, you can get the latest version with the following:

# install kubectl plugin for asdf
asdf plugin-add kubectl \
  https://github.com/asdf-community/asdf-kubectl.git
asdf install kubectl latest

# fetch latest kubectl 
asdf install kubectl latest
asdf global kubectl latest

# test results of latest kubectl 
kubectl version --short --client 2> /dev/null

This should show something like:

Client Version: v1.27.1
Kustomize Version: v5.0.1

Also, create directory to store Kubernetes configurations that will be used by the KUBECONFIG env variable:

mkdir -p $HOME/.kube

Setup Environment Variables

These environment variables will be used throughout this project. If opening up a browser tab, make sure to set the environment variables accordingly.

# variables used to create EKS
export AWS_PROFILE="my-profile" # CHANGEME
export EKS_CLUSTER_NAME="my-cluster" # CHANGEME
export EKS_REGION="us-west-2"
export EKS_VERSION="1.25"

# KUBECONFIG variable
export KUBECONFIG=$HOME/.kube/$EKS_REGION.$EKS_CLUSTER_NAME.yaml

# variables used in automation
export ROLE_NAME="${EKS_CLUSTER_NAME}_EBS_CSI_DriverRole"
export ACCOUNT_ID=$(aws sts get-caller-identity \
  --query "Account" \
  --output text
)
export ACCOUNT_ROLE_ARN="arn:aws:iam::$ACCOUNT_ID:role/$ROLE_NAME"

Provision EKS cluster

A minimal EKS cluster can be provisioned with the following command:

eksctl create cluster \
  --region $EKS_REGION \
  --name $EKS_CLUSTER_NAME \
  --version $EKS_VERSION

Once this finished in about 20 minutes, install a kubectl version that matches the server version:

# fetch exact version of Kubernetes server (Requires GNU Grep)
VER=$(kubectl version --short \
  | grep Server \
  | grep -oP '(\d{1,2}\.){2}\d{1,2}'
)

# setup kubectl tool
asdf install kubectl $VER
asdf global kubectl $VER

Also, check the status of the worker nodes and applications running on Kubernetes.

kubectl get nodes
kubectl get all --all-namespaces

This should show something like the following.

Add OIDC Provider Support

The EKS cluster has an OpenID Connect (OIDC) issuer URL associated with it. To use AWS IRSA, an IAM OIDC provider must exist for the cluster’s OIDC issuer URL.

You can set this up with the following command:

eksctl utils associate-iam-oidc-provider \
  --cluster $EKS_CLUSTER_NAME \
  --region $EKS_REGION \
  --approve

Associate Service Account

This next step will setup the necessary identities and permissions that allow ebs-csi driver to access the required privileges needed to mount EBS volumes.

# AWS managed policy for CSI driver SA to make EBS API calls
POLICY_ARN="arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy"

# AWS IAM role bound to a Kubernetes service account
eksctl create iamserviceaccount \
  --name "ebs-csi-controller-sa" \
  --namespace "kube-system" \
  --cluster $EKS_CLUSTER_NAME \
  --region $EKS_REGION \
  --attach-policy-arn $POLICY_ARN \
  --role-only \
  --role-name $ROLE_NAME \
  --approve

This command eksctl create iamserviceaccount does the following automation:

1. Create an IAM Role with a trust policy federated by the OIDC provider associated with the EKS cluster
2. Attach the policy needed to grant required access for AWS EBS APIs.
3. Create a Service Account with appropriate metadata annotations that will associate it back to the corresponding IAM Role. This part is disabled due the --role-only argument. This will be handled by either the EKS add-on or the helm chart later in this article.

Install AWS EBS CSI driver

Select one of two methods you can use for this installation:

1. Method A: Install using EKS addon feature
2. Method B: Install using Helm chart

Pick on of these methods and follow the instructions in that secction.

Method A: Install using EKS addon feature

AWS has introduced an EKS addon feature to help with better lifecycle management. There are naturally some trade-offs, as this is less consistent, for everyone understands Helm charts.

You can use eksctl to automate adding the aws-ebs-csi-driver addon.

# Create Addon
eksctl create addon \
  --name "aws-ebs-csi-driver" \
  --cluster $EKS_CLUSTER_NAME \
  --region=$EKS_REGION \
  --service-account-role-arn $ACCOUNT_ROLE_ARN \
  --force

# Get status of the driver, must be STATUS=ACTIVE
eksctl get addon \
  --name "aws-ebs-csi-driver" \
  --region $EKS_REGION \
  --cluster $EKS_CLUSTER_NAME

You can check on the running pods with the following command:

kubectl get pods \
  --namespace "kube-system" \
  --selector "app.kubernetes.io/name=aws-ebs-csi-driver"

This should show something like:

Once the pods are up and the addon status is set you ACTIVE, you can go ahead to the section on setting up the storage class and testing out storage with Dgraph.

Method B: Install using the Helm chart

The helm chart method may be more familar as helm charts are more ubiquitous across the Kubernetes ecosystem.

IRSA_KEY="eks\\.amazonaws\\.com/role-arn"

helm repo add aws-ebs-csi-driver \
  https://kubernetes-sigs.github.io/aws-ebs-csi-driver
helm repo update

helm upgrade \
  --install aws-ebs-csi-driver \
  --namespace kube-system \
  --set controller.serviceAccount.annotations.$IRSA_KEY=$ACCOUNT_ROLE_ARN \
  aws-ebs-csi-driver/aws-ebs-csi-driver

You can check on the running pods with the following command:

kubectl get pods \
  --namespace "kube-system" \
  --selector "app.kubernetes.io/name=aws-ebs-csi-driver"

This should show something like the following.

After this the pods are up, you can do to the next section for setting up the storage class and then test the new storage with Dgraph distributed graph database.

Setup Storage Class for the AWS EBS CSI driver

Now that the EBS CSI driver is installed, we can create a storage class that uses this new storage.

Create a file named storageclass.yaml with the following contents below:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

To apply this, run the following command:

kubectl apply --filename storageclass.yaml

You can check out the results of this with:

kubectl get storageclass

This should show something like the following

Dgraph

Dgraph is a highly performant distributed graph database. Obviously, since it is a database, it requires stroage. To install Dgraph and use the storage we just setup, run the following commands.

helm repo add "dgraph" "https://charts.dgraph.io"
helm repo update

helm install "my-release" dgraph/dgraph \
  --namespace "dgraph" \
  --create-namespace \
  --set zero.persistence.storageClass=ebs-sc \
  --set alpha.persistence.storageClass=ebs-sc  

You can verify the resources created with

kubectl get all --namespace "dgraph"

This should show something like the following

Cleanup

Remove Dgraph

You can delete Dgraph and EBS volumes with the following command.

helm delete "my-release" --namespace "dgraph"
kubectl delete pvc --selector "release=my-release" --namespace "dgraph"

⚠️ It is important to delete the pvc (persistent volume claims) before deleting the EKS cluster. If this is not done, then there will be leftover EBS volumes that incur futher costs. 💵 💶 💴 💷

Remove EKS Addon (only if Method A)

If you installed the EBS CSI driver using the EKS Addon, then you can delete the the addon with the following below. Make sure there are no persistent volumes using use the storage class, e.g. kubectl get pvc --all-namespaces | grep -E 'STORAGECLASS|ebs-sc'.

eksctl delete addon \
  --name "aws-ebs-csi-driver" \
  --cluster $EKS_CLUSTER_NAME \
  --region $EKS_REGION

Remove IAM Role

You can delete the associated IAM role as well with the following comand:

eksctl delete iamserviceaccount \
  --name "ebs-csi-controller-sa" \
  --namespace "kube-system" \
  --cluster $EKS_CLUSTER_NAME

Delete EKS Cluster

Finally, now EKS cluster can be deleted.

eksctl delete cluster \
  --region $EKS_REGION \
  --name $EKS_CLUSTER_NAME

Resources

IAM OIDC and IRSA documentation

• Creating an IAM OIDC provider for your cluster

EKS Addons documentation

• Amazon EKS add-ons
• Introducing Amazon EKS add-ons: lifecycle management for Kubernetes operational software

EBS CSI documentation

• Amazon EBS CSI
• Creating an IAM OIDC provider for your cluster

External guides and videos

• EKS Storage | Install Amazon EBS CSI driver & KMS (video)
• Workshop: Amazon EBS CSI (Container Storage Inerface) Driver (archived, deprecated)

Conclusion

In the previous guide, I demonstrated the minimalist setup for EKS cluster with storage support for demo test environments. This article expands on this, keeping the minimal steps for demonstration purposes, but expanding on how to more securely install EBS CSI Driver applying concepts like PoLP.

The major takeaways beyond installing EBS CSI driver are:

• IRSA (IAM roles for service accounts) facility to limit access to AWS cloud resources
• EKS Add-ons feature vs Helm charts to extend the EKS cluster

The tool eksctl is a quick and easy way to get started with EKS including some more advance features, like setting up an IAM role with federated trust to an OIDC provider. This tool is robust enough to a point that I am confident that some will use in production environments. The alternative with aws or terraform commands would make this article at least five times longer.