AKS with gRPC and ingress-nginx

Using gRPC with ingress-nginx add-on with AKS

Updated: 2021–08–30 multi-line code to gist as hard to copy from Medium

In previous articles in this series, I covered how to publish end points for applications that are deployed on , ultimately detailing how to use Ingress resources with automation support for certificates and updating DNS records. This article pick that up, but shows how to support , a popular protocol for efficient web APIs.

In supporting web interfaces, traditionally with , that is ultimately a (Create, Read, Update, Delete) interface using protocol. Part of using such web interfaces requires converting data-structures to and from a format, a process called and . This process with text files in a data structure () is not particularly efficient.

() is an alternative that provides significant performance gains for web interfaces at least in two regards: more highly performant and serialization using binary format.

This article demonstrates how to configure a ingress resource for with . The following components will be created:

Articles in this Series

  1. : service with LoadBalancer type
  2. : ingress (HTTP)
  3. : ingress (HTTPS)
  4. : ingress (GRPC and HTTPS)

Previous Article

In the previous article showed how to automate creation of certificates with .

Requirements

These are some logistical and tool requirements for this article:

Registered domain name

will require use encryption with certificates, so that it can route traffic between (HTTP/2) and HTTPS (HTTP/1.1). A certificate that are issued by a trusted will require you to own a public domain name, which can be purchased from a provider for about $2 to $20 per year.

A fictional domain of example.com will be used as an example. Thus depending on the examples used, there would be, for example, hello.example.com, ratel.example.com, and alpha.example.com.

Required tools

These tools are required for this article:

  • (az): command line tool that interacts with Azure API
  • (kubectl): command line tool that interacts with Kubernetes API
  • (helm): command line tool for “templating and sharing Kubernetes manifests” that are bundled as Helm chart packages.
  • plugin: allows you to see the changes made with helm or helmfile before applying the changes.
  • (helmfile): command line tool that uses a “declarative specification for deploying Helm charts across many environments”.

Optional tools

I highly recommend these tools:

  • (sh) such as (bash) or (zsh): these scripts in this guide were tested using either of these shells on macOS and Ubuntu Linux.
  • (curl): tool to interact with web services from the command line.
  • (grpcurl): a tool to interact with services from the command line.
  • (jq): a JSON processor tool that can transform and extract objects from JSON, as well as providing colorized JSON output greater readability.
  • (python) and (pip) to use Dgraph python script for interaction. Recommend using and to manage requirements.

Project setup

Below is a file structure that will be used for this article to keep things consistent and referenceable.

Project file structure

The following structure will be used:

~/azure_ingress_nginx_grpc/
├── env.sh
├── examples
│ ├── dgraph
│ │ ├── data
│ │ │ ├── getting_started.py
│ │ │ ├── sw.nquads.rdf
│ │ │ └── sw.schema
│ │ └── helmfile.yaml
│ └── yages
│ └── helmfile.yaml
├── helmfile.yaml
└── issuers.yaml

With either or , you can create the file structure with the following commands:

These instructions from this point will assume that you are in the $HOME/azure_ingress_nginx_grpc directory, so when in doubt:

cd ~/azure_ingress_nginx_grpc

Project environment variables

Setup these environment variables below to keep things consistent amongst a variety of tools: , , , , , , .

If you are using a , you can save these into a script and source that script whenever needed. Copy this source script and save as env.sh:

After composing this script, source it so that variables can be used in the reset of the steps.

source env.sh

Azure components

Similar to previous articles, both and zone cloud resources, and then authorize worker nodes to access , so that any pod running on can make the required DNS updates.

For simplicity, you can use the following steps below to create the required resources. If you use your own scripts, such as Terraform, be sure to enable or with the cluster.

Cloud resources

You can create and cloud resources with the following commands:

You will need to transfer domain management to for root domain like example.com, or if you are using sub-domain like dev.example.com, you’ll need to update DNS namespace records to point to name servers. This process is fully detailed as well as how to provision the equivalent with Terraform in article.

For a more robust script on provisioning , see article.

Verify AKS and KUBCONFIG

Verify that the AKS cluster was created and that you have a KUBCONFIG that is authorized to access the cluster by running the following:

source env.sh # fetch KUBECONFIGkubectl get all --all-namespaces

The final results can should look something like this:

Provisioned AKS (kubenet network plugin)

NOTE: Regarding security around access, this setup with KUBECONFIG grants full unfettered access to anyone that has the configuration. This may be fine for single users development environments, but in a shared environment, you will want to secure this further, such as using to secure access to .

Authorizing access Azure DNS

Both and will need access to the zone. This can be done through by associating a role that grants access to the zone to the installed on the node pool workers.

Once completed, this allows all pods running on the cluster to update records on the zone.

NOTE: In regards to security, allowing any pod to change DNS records could be dangerous, especially if any of the records were used for production infrastructure. Currently in preview, has a upcoming feature called s that can secure access at the pod level instead of the node level. This can be used to allow access for only and services.

You can grant access to zone using the following commands:

About Managed Identities on AKS

A is a wrapper around to make management simpler. Essentially, they are mapped to a resource, so that when the resource no longer exists, the associated will be removed.

A service principal is sent to , which if authorized based on the role assignment, will grant a ticket or token. This token is then used to access the service explicitly for the zone we specified, e.g. example.com.

Managed Identity authorized to access Azure DNS

Kubernetes components

The add-ons can be installed with the following script below.

Install the addons

Copy this script below and save as helmfile.yaml:

Once ready, simply run:

############ 
# fetch environment variables
# AZ_TENANT_ID, AZ_SUBSCRIPTION_ID, AZ_DNS_DOMAIN
############################################
source env.sh
############
# deploy k8s add-ons
############################################

helmfile
apply

Install clusterissuers

Copy the following and save as issuers.yaml:

There will need to be a few seconds before the pods are ready and online. When ready, run this:

############ 
# fetch environment variables
# ACME_ISSUER_EMAIL, AZ_DNS_DOMAIN, AZ_SUBSCRIPTION_ID
############################################
source env.sh
############
# deploy cert-manager ACME issuers
############################################
helmfile --file issuers.yaml apply

Verify Addons and Cluster Issuer resources

Once this process is completed you can view the results with:

kubectl get all,clusterissuer --namespace kube-addons

This should something like the following:

Kube addon deploy (cert-manager, external-dns, ingress-nginx)

GRPC ingress example: Yages

This example uses a small service called ().

Copy the following below and save it as examples/yages/helmfile.yaml:

Deploy YAGES

This has three embedded Kubernetes resources: service, ingress, and deployment for the application. You can deploy the following into the yages namespace using the letsencrypt-prod issuer with the following:

############ 
# fetch environment variables
# ACME_ISSUER, AZ_DNS_DOMAIN
############################################

source
env.sh
############
# deploy yages
############################################

helmfile
-f examples/yages/helmfile.yaml apply

Verify YAGES

After about 3 minutes for the certificate to be ready, you can run the following to verify the resources are deployed and ready:

kubectl get all,ing,certificate --namespace yages

The results should look something like this:

yages deployment

Testing gRPC with YAGES

Make sure that the certificate Ready state is true.

kubectl get certificate --namespace yages --watch

Once ready we can test the service with using the following commands:

NOTE: This assumes that ACME_ISSUER is set to letsencrypt-prod. If this is set to letsencrypt-staging for certificates issued by a private CA, use the grpcurl --insecure argument.

GRPC ingress example: Dgraph

is a distributed graph database and has a helm chart that can be used to install into a cluster. You can use either helmfile or helm methods to install .

Dgraph and Ratel infrastructure

This example will deploy three endpoints using 2 × ingress resources, one for traffic and one for traffic. Both use the same certificates, so in the background these will only be created once for both ingress resources. Ultimately all of these rules from all ingresses will be added to the same configuration () in the ingress-controller pods.

This will allow the following access (swapping out example.com for the domain name you are using):

  • dgraph.example.com: backend graph database through GRPC (port 9080)
  • alpha.example.com: backend graph database through HTTPS (port 8080)
  • ratel.example.com: the graphical user interface client ( application) (port 80)

Securing Dgraph (optional)

In a production scenario, public endpoints should be secured, especially for a backend database, but in order the keep things simple for the article, the endpoint will not be secured. Normally, this would be done with a or even better, only have the endpoints exposed internally and then use a or to access the backend service.

Some level of security can be added on the service itself by adding an allow list (also called a whitelist):

This value DG_ALLOW_LIST will be used later during deployment. If this is not set, it will default to allow the whole public Internet access to the database server.

Dgraph Deploy Code

Copy the file below and save as examples/dgraph/helmfile.yaml:

Deploy Dgraph Services and Ingresses

The will deploy the Dgraph helm chart for several resources including Dgraph Ratel UI deployment and service, Dgraph Alpha and Dgraph Zero statefulsets, pvc, service, headless service, a configmap, as well as some custom manifests to deploy two ingress resources: one for GRPC traffic and one for HTTP traffic.

When this can be deployed with:

############ 
# fetch environment variables
# ACME_ISSUER, AZ_DNS_DOMAIN
############################################

source
env.sh
############
# deploy yages
############################################

helmfile
-f examples/dgraph/helmfile.yaml apply

Verify deployed Dgraph resources

After a few minutes, you can verify the resources with the following command:

kubectl get all,ing,certificate --namespace dgraph

This should look something like the following:

Dgraph + ingress + certificate deployment

The certificate may take about a minute before it is ready. You can monitor it with:

kubectl get certificate --namespace dgraph --watch

Also the pods may take a about a minute to before they are in a ready state. This can be monitored with:

kubectl get pods --namespace dgraph --watch

Verify Basic gGRPC

For getting off the ground to determine if in (or ), the tool grpcurl can be used to test this:

The response should be something like:

{
"tag": "v21.03.0"
}

Upload Data and Schema

There are some scripts adapted from tutorials that you can download.

First, and save the following at examples/dgraph/data/getting_started.py:

The above script is based on and and , and is limited to just working with only certificates issued from publicly trusted CAs. This can be easily adapted for insecure (no ) traffic, or with private certificates including mutual where a client certificate is required. For further information, check out the code repository:

This above python script does have a few module requirements, which you can install with from your 3 environment:

pip install pydgraph
pip install certifi

Then next step is to download the schema and data files, and then run the script:

This will upload the schema and data file, so that we can run queries.

Connect to Ratel UI

After a few moments, you can check the results https://ratel.example.com (substituting example.com for your domain).

In the dialog for Dgraph Server Connection, configure the domain, e.g. https://alpha.example.com (substituting example.com for your domain)

Test Using the Ratel UI

In the Ratel UI, paste the following query and click run:

You should see something like this:

Cleanup the Project

You can cleanup resources that can incur costs with the following:

Remove the Azure Resources

This will remove the Azure resources:

############ 
# Delete the AKS cluster
############################################

az
aks delete \
--resource-group $AZ_RESOURCE_GROUP \
--name $AZ_CLUSTER_NAME
############
# Delete the Azure DNS Zone
############################################

az
network dns zone delete \
--resource-group $AZ_RESOURCE_GROUP \
--name $AZ_DNS_DOMAIN

Resources

Here are some articles and links I have come across on topics while making this article.

Blog Source Code

Azure Resources

Kubernetes Addons

Services used in ingress examples

Ingress-Nginx and GRPC

Protobuffers and gRPC

Python (gRPC and TLS)

In doing this guide, I needed to figure out how to communicate securely with and in . Most documentation covered using only or certificates issued by a non-trusted private . As we are using certificates from a publicly trusted through through , it was hard to find material on this. Below are some links that may be useful, should find yourself on the same journey.

Conclusion

This article was a challenge in two pieces, generally getting to work with existing services that may still be needed, and testing itself, where the requirement go beyond a simple curl command. The underlying protocol is different, as it requires serialization through and is a different, not stateless like , so load balancer have to load balance on sessions, rather than transactions.

The python script (or any other language for that matter) was a little challenging, as most services that are will almost never be available publicly, at least without some form of authentication. Typically, for internal private communication, such as microservice-to-microservice, or microservice-to-graphdb, private certificates are perfectly fine.

So I think this concludes the series of these articles. I could maybe document how to further lock-down or these services, but likely I can see trying out some other options for with some different , like , , , or , or jumping into service meshes like or .

Linux NinjaPants Automation Engineering Mutant — exploring DevOps, o11y, k8s, progressive deployment (ci/cd), cloud native infra, infra as code

Linux NinjaPants Automation Engineering Mutant — exploring DevOps, o11y, k8s, progressive deployment (ci/cd), cloud native infra, infra as code