Building AKS with Terraform

Provision an AKS Kubernetes Cluster with Terraform

This article covers provisioning a basic high-availability Kubernetes cluster with AKS (Amazon Kubernetes Service) using Terraform with the azurerm provider.

Previously, I covered how to do this using a shell script wrapper around Azure CLI. For this article, all cloud resources (Azure and Kubernetes) will be created with terraform.

NOTE: Medium code blocks have been challenging (nearly impossible) to copy text, so to help readers, I have been moving code snippets to gists.

Requirements

The Tools

These are the baseline tools needed to work with Azure and Kubernetes.

• Azure CLI tool (az): command line tool that interacts with Azure API
• Kubernetes client tool (kubectl): command line tool that interacts with Kubernetes API
• Terraform (terraform): command tool to provision Azure cloud resources and deploy Kubernetes applications.

Project Setup

The following file structure will be used:

~/azure_basic/
├── demos
│   └── hello-kubernetes
│       ├── main.tf
│       ├── provider.tf
│       └── terraform.tfvars
├── main.tf
├── provider.tf
└── terraform.tfvars

You can create this with the following commands:

For the rest of the article, commands will be executed from the $HOME/azure_basic directory.

cd ~/azure_basic

Provision Azure resources

This process will provision an AKS cluster, where Azure will then create a new resource group and provision cloud resources needed for the AKS cluster.

For a basic HA AKS cluster, two modules will be used: group and aks. These will be fetched from the blog source code repository.

Create the Terraform scripts

Create the file provider.tf with the following contents:

Create the file main.tf with the following contents:

Create the variable definition

Create terraform.tfvars with the following content:

These are some example values for this project that you should change as appropriate. If you have an existing resource group that you wish to use, change create_group to false, otherwise, Terraform will attempt to create it.

Provision AKS with Terraform

When ready, run the following commands:

NOTE: Currently there is no direct method to create dependencies between modules in Terraform. Thus if you need a resource created before other modules are executed, then you will have to manually orchestrate this with the -target command-line argument.

Interact with the Kubernetes cluster

In order to interact with the cluster, we need to setup a KUBECONFIG. Run these commands to populate a configuration with credentials and point environment variable KUBECONFIG to that file.

After this run the command to see the components installed on a vanilla cluster:

kubectl get all --all-namespaces

This should show something like the following:

Exploring Kubernetes networking: Kubenet

Run the following command to see the nodes and pods and their corresponding IP addresses:

This should show something like the following:

This output shows both nodes and daemonset privileged pods running on the Azure VNET created with the cluster, while other pods are on an overlay networks created by kubenet network plugin.

Exploring Kubernetes networking: Routes

The kubenet network plugin will need external routes configured. You can see this in action by looking are the Azure virtual network traffic routing with the following command:

This should show something like the following:

Demo: hello-kubernetes

For this demo application, hello-kubernetes will be used, which will display the names of the pods and nodes.

Create the provider script

For the kubernetes provider, you need to specify where to access credentials, such as KUBECONFIG credentials. As we created the cluster with Azure, we can use the azurerm provider to fetch the credentials.

Create the file demos/hello-kubernetes/provider.tf with the following contents:

Create the main script

The main part of the script will deploy two Kubernetes resources: service and deployment.

Create the file demos/hello-kubernetes/main.tf with the following contents:

Create the variable definition

Create demos/hello-kubernetes/terraform.tfvars with the following content:

Deploy hello-kubernetes

When ready, run the following commands to deploy hello-kubernetes demo and verify the results.

NOTE: Embedding is currently not working. Refer to this link:

This should show something like the following:

You can access the one of the pods using port-forward:

Afterward, you should see something like this with http://localhost:8080:

Cleanup

Kubernetes: hello-kubernetes

If you would like to just delete hello-kubernetes application, you can do the following:

pushd demos/hello-kubernetes && terraform destroy && popd

AKS Cluster

If you just want to delete the Kubernetes cluster and associated resources, e.g. load balancer, managed identity, VMSS, and NSG, then run this command:

terraform destroy

Resources

These are resources that I have discovered along the way when using Terraform and AKS.

Blog Source Code

• https://github.com/darkn3rd/blog_tutorials/tree/master/kubernetes/aks/series_0_provisioning/terraform/0_basic

Tutorials

• Create a Kubernetes cluster with Azure Kubernetes Service using Terraform (Azure Documentation)
• Provision an AKS Cluster (Azure) (Hashicorp Learn)
• Getting started with Terraform and Kubernetes on Azure AKS (LearnK8S)

Conclusion

This is a very basic tutorial covering how to get started with Terraform for Azure, AKS, and Kubernetes. This tutorial doesn’t go deep into the intricacies of provisioning AKS, as this will be covered in the source code supplement, should you want to delve further into this topic.

The Takeaways

Main takeaways:

• Provision AKS using Terraform (azurerm provider)
• Deploy K8S applications using Terraform (kubernetes provider)

Extra takeaways are exposure to:

• network plugins with the default kubenet
• Azure CLI (az) command with JMESPath to gather information
• kubectl command with JSONPath to gather information

Where to go next?

For Azure and AKS, there are more advance scenarios when configure AKS to work with Azure DNS, Azure Container Registry, Azure Key Vault, Azure Blob Storage, and other resources. These require setting up identities or service principals, which are created from Azure Active Directory , which is Kerberos + LDAP under the hood.

On the Kubernetes side, Terraform is becoming popular for deploying applications and other configurations. For advance scenarios where you need to use templates, Terraform is not very robust in this area.

As an alternative you can use Helm charts or orchestrate charts with Helmfile. These can be used within Terraform using Helm provider and Helmfile providers.

Thanks for following along.