AKS with ingress-nginx

Using ingress-nginx add-on with Azure LB and AKS

This article covers adding an ingress controller called ingress-nginx to AKS (Azure Kubernetes Service). An ingress is a reverse proxy that allows routing web traffic to appropriate targeted services using a single load balancer.

Articles in the Series

These articles are part of a series, and below is a list of articles in the series.

  1. AKS with ingress-nginx: ingress (HTTP)
  2. AKS with cert-manager: ingress (HTTPS)
  3. AKS with GRPC and ingress-nginx: ingress (GRPC and HTTPS)

Previous Articles

In the previous article, I wrote about how to use external-dns with Azure DNS to configure DNS address records automatically when deploying Kubernetes application. This is configured through a service annotation.

Requirements

These are some logistical and tool requirements for this article:

Registered domain name

As this tutorial uses a public domain name, so you will need to purchase this from somewhere to follow everything in this tutorial. This generally costs about $2 to $20 per year.

  • skip using domains altogether, and instead use HTTP paths instead, such as /hello, /ratel, /alpha depending on the examples used.

Required Tools

  • Kubernetes client tool (kubectl): command line tool that interacts with Kubernetes API
  • Helm (helm): command line tool for “templating and sharing Kubernetes manifests” that are bundled as Helm chart packages.
  • helm-diff plugin: allows you to see the changes made with helm or helmfile before applying the changes.
  • Helmfile (helmfile): command line tool that uses a “declarative specification for deploying Helm charts across many environments”.

Optional Tools

  • curl (curl): tool to interact with web services from the command line.
  • jq (jq): a JSON processor tool that can transform and extract objects from JSON, as well as providing colorized JSON output greater readability.

Project Setup

As this project has a few moving parts (Azure DNS, AKS, ingress-nginx, ExternalDNS) with example applications Dgraph and hello-kubernetes, these next few will help keep things consistent.

Project File Structure

The following structure will be used:

~/azure_ingress_nginx/
├── env.sh
├── examples
│ ├── dgraph
│ │ └── helmfile.yaml
│ └── hello
│ └── helmfile.yaml
└── helmfile.yaml
cd ~/azure_ingress_nginx

Project Environment Variables

Setup these environment variables below to keep things consistent amongst a variety of tools: helm, helmfile, kubectl, jq, az.

source env.sh

Resource Group

In Azure, resources are organized under resource groups.

az group create -n $AZ_RESOURCE_GROUP -l $AZ_LOCATION

Azure Cloud Resources

For simplicity, you can create the resources needed for this project with the following:

Authorizing access Azure DNS

We need to allow access to the Managed Identity installed on VMSS node pool workers to the Azure DNS zone. This will allow any pod running a Kuberentes worker node to access the Azure DNS zone.

ExternalDNS using Managed Identity
/subscriptions/<subscription id>/resourceGroups/<resource group name>/providers/Microsoft.Network/dnszones/<zone name>/

Installing Addons

Now that the infrastructure is provisioned, we can install the required Kubernetes add-ons ingress-nginx and external-dns, and configure these to use the Azure DNS Zone for DNS record updates.

source env.sh && helmfile apply

Example using hello-kubernetes

The hello-kubernetes is a simple application that prints out the pod names. The script below includes an embedded ingress manifest that includes host name configuration value. This will be scanned by ExternalDNS, which will then trigger automation to configure an address record on the Azure DNS zone.

Example: hello-kubernetes
source env.sh
pushd
./examples/hello/
helmfile
apply
popd

Verify Hello Kubernetes is deployed

kubectl get all,ing --namespace hello

Verify Hello Kubernetes works

After a few moments, you can check the results http://hello.example.com (substituting example.com for your domain).

Example using Dgraph

Dgraph is a distributed graph database and has a helm chart that can be used to install Dgraph into a Kubernetes cluster. You can use either helmfile or helm methods to install Dgraph.

Example: Dgraph Alpha + Dgraph Ratel

Dgraph Endpoint Security

There are two applications that are connected to the public Internet through the ingress: Dgraph Ratel and Dgraph Alpha. The Dgraph Ratel is just a client (React), so given its limitations, has a low risk to be exposed. The Dgraph Alpha service is another story, as this is the backend database and should never be exposed to the public Internet without some sort of restriction, such as network security group that limits in-bound traffic to your home office.

  • Deploy an ingress that uses an internal load balancer private subnets, and use this ingress for the Dgraph endpoint.
  • Use a SSH jump host or a VPN to access the private subnets for administrative or development purposes.

Dgraph Alpha Security

We can apply some security on the Dgraph Alpha service itself by adding an allow list (also called a whitelist):

# get AKS pod and service IP addresses
DG_ALLOW_LIST
=$(az aks show \
--name $AZ_CLUSTER_NAME \
--resource-group $AZ_RESOURCE_GROUP | \
jq -r '.networkProfile.podCidr,.networkProfile.serviceCidr' | \
tr '\n' ','
)
# append home office IP address
MY_IP_ADDRESS=$(curl --silent ifconfig.me)
DG_ALLOW_LIST="${DG_ALLOW_LIST}${MY_IP_ADDRESS}/32"
export DG_ALLOW_LIST

Deploy Dgraph

Copy the file below and save as examples/dgraph/helmfile.yaml:

source env.sh
pushd
examples/dgraph/
helmfile apply
popd

Verify Dgraph is deployed

Check that the services are running:

kubectl get all,ing --namespace dgraph

Verify Dgraph Alpha health check

Verify that the Dgraph Alpha is accessible by the domain name (substituting example.com for your domain):

curl --silent http://alpha.example.com/health | jq

Upload Data and Schema

There are some scripts adapted from tutorials https://dgraph.io/docs/get-started/ that you can down:

PREFIX=gist.githubusercontent.com/darkn3rd
RDF_GIST_ID=398606fbe7c8c2a8ad4c0b12926e7774
RDF_FILE=e90e1e672c206c16e36ccfdaeb4bd55a84c15318/sw.rdf
SCHEMA_GIST_ID=b712bbc52f65c68a5303c74fd08a3214
SCHEMA_FILE=b4933d2b286aed6e9c32decae36f31c9205c45ba/sw.schema
curl -sO https://$PREFIX/$RDF_GIST_ID/raw/$RDF_FILE
curl
-sO https://$PREFIX/$SCHEMA_GIST_ID/raw/$SCHEMA_FILE
curl -s "alpha.$AZ_DNS_DOMAIN/mutate?commitNow=true" \
--request POST \
--header "Content-Type: application/rdf" \
--data-binary @sw.rdf | jq
curl -s "alpha.$AZ_DNS_DOMAIN/alter" \
--request POST \
--data-binary @sw.schema | jq

Connect to Ratel UI

After a few moments, you can check the results http://ratel.example.com (substituting example.com for your domain).

Test Using the Ratel UI

In the Ratel UI, paste the following query and click run:

{
me(func: allofterms(name, "Star Wars"),
orderasc: release_date)
@filter(ge(release_date, "1980")) {
name
release_date
revenue
running_time
director {
name
}
starring (orderasc: name) {
name
}
}
}

Cleanup Dgraph resources

You can remove Dgraph resources, load balancer, and external disks with the following:

helmfile --file examples/dgraph/helmfile.yaml delete
kubectl
delete pvc --namespace dgraph --selector release=demo

Cleanup the Project

You can cleanup resources that can incur costs with the following:

Remove External Disks

Before deleting AKS cluster, make sure any disks that were used are removed, otherwise, these will be left behind an incur costs.

kubectl delete pvc --namespace dgraph --selector release=demo

Remove the Azure Resources

This will remove the Azure resources:

az aks delete \
--resource-group $AZ_RESOURCE_GROUP \
--name $AZ_CLUSTER_NAME
az network dns zone delete \
--resource-group $AZ_RESOURCE_GROUP \
--name $AZ_DNS_DOMAIN

Resources

Blog Source Code

Conclusion

This article shows how to use an ingress controller (ingress-nginx) combined with external-dns for automatic record creation. This was a follow-up article where I showed previously how to do the same thing with a service resource.

  • A service of type LoadBalancer requires a load balancer per endpoint.
  • The external load balancer are typically Layer 4 (TCP), where an ingress is Layer 7 (HTTP)
  • How to use managed identities to allow pods running on AKS worker nodes (VMSS) to access Azure resources.

Linux NinjaPants Automation Engineering Mutant — exploring DevOps, o11y, k8s, progressive deployment (ci/cd), cloud native infra, infra as code