GKE with gRPC and ingress-nginx

Using gRPC with ingress-nginx add-on with GKE

About ingress-nginx

About Dgraph

Components

📔 NOTE: This was tested on following below and may not work if versions are significantly different.* Kubernetes API v1.22
* kubectl v1.22
* gcloud 394.0.0
* external-dns v0.12.2
* cert-manager v1.9.1
* ingress-nginx 1.3.0
* Docker 20.10.17
* Dgraph v21.03.2

Requirements

Accounts

  • To follow the steps in this article, you will need to have a registered domain and forward DNS queries to the Cloud DNS name servers.
    Consult documentation from the registrar for your domain.
    This tutorial will use example.com as an example domain.
  • Google Cloud account a billing account and project setup. Google is offering a 90-day $300 free trial (Aug 2022) that is sufficient for this article. See https://cloud.google.com/free.
  • Set ACME_ISSUER_NAME=letencryupt-staging before deploying any ingress.
  • Edit edit/etc/hosts (or equivalent) to match the DNS records, local DNS cache, or configuring DNS client to point to Cloud DNS for the search domain for that domain.
  • When accessing a service through the web like Ratel, you will need to click on add an exception when prompted for an untrusted website.
  • When using the curl command, you will have to use the -k option, e.g. curl -k.

Knowledge

  • Basic knowledge of using Google Cloud SDK to configure access, setup a project, and provision resources.
  • Basic shell scripting knowledge including things like setting up environment variables. Python is useful for understanding the load_data.py script, but not required.
  • Basic Kubernetes using kubectl command to deploy applications and setup configuration with the KUBECONFIG environment variable. Understanding Kubernetes resources types like Deployment, StatefulSet, ReplicaSets, Pods, Service(L4), Ingress(L7) are useful.
  • Basic networking knowledge of TCP (Layer 4 vs Layer 7), knowledge about HTTP/2 vs HTTP/1.1 protocols, and exposure to TLS vs SSL certificates.
  • Understanding of load balancers and reverse proxies, and routing based on ports, Virtual Host, URL Paths.

Tools (Required)

Images of tool icons: gcloud, kubectl, helm, helm-diff, helmfile, docker
  • Google Cloud SDK (gcloud command) to interact with Google Cloud
  • Kubernetes client (kubectl command) to interact with Kubernetes
  • Helm (helm command) to install Kubernetes packages
  • helm-diff plugin to see differences about what will be deployed.
  • helmfile (helmfile command) to automate installing many helm charts
  • Docker Engine (docker command) to automate running pydgraph client and all its dependencies locally.

Tools (Recommended)

Images of tool icons: curl, grpcurl, zsh, bash, jq
  • POSIX shell (sh) such as GNU Bash (bash) or Zsh (zsh): these scripts in this guide were tested using either of these shells on macOS and Ubuntu Linux.
  • GNU stream-editor (sed) and GNU grep (grep): scripts were tested with these tools and the macOS or BSD equivalents may not work.
  • curl (curl): tool to interact with web servers from the command line.
  • jq (jq): a JSON processor tool that can transform and extract objects from JSON, as well as providing colorized JSON output greater readability.
  • gprcurl (gprcurl): tool to interact with gRPC servicers from the command line.

Project Setup

Directory structure

~/projects/ingress-nginx-grpc
├── dgraph
│ └── helmfile.yaml
├── kube-addons
│ ├── helmfile.yaml
│ └── issuers.yaml
└── ratel
└── helmfile.yaml
mkdir -p ~/projects/ingress-nginx-grpc/{dgraph,ratel,kube-addons} 
cd ~/projects/ingress-nginx-grpc
touch {dgraph,kube-addons,ratel}/helmfile.yaml \
kube-addons/issuers.yaml

Environment variables

USER="darkn3rd"
ID="7af3da347073b0ddf20fd7fa0c4e69c7"
VERS="a3ac0c761e49c2ca8cd88f2e0d75d04dd3f4ed1c"
FILE="validate.sh"
URL=https://gist.githubusercontent.com/$USER/$ID/raw/$VERS/$FILE
curl -s $URL | bash -s --

Google project setup

Provision Cloud Resources

Cloud DNS

ns-cloud-d1.googledomains.com.
ns-cloud-d2.googledomains.com.
ns-cloud-d3.googledomains.com.
ns-cloud-d4.googledomains.com.

Google Kubernetes Engine

⚠️ NOTE: Though this Kubernetes cluster is secure in as far as principal of least privilege with identity principals (Google service accounts) for securing access to cloud resources, the master nodes and worker nodes are  accessible from the public Internet.For further security, such as  production environments, you may  consider using private masters nodes, which require some form of jump host or VPN to access them, as also having worker nodes on a private and public networks (subnets), so that you have further control of what endpoints should be explicitly exposed to the public Internet.Additionally, using a CNI plugin that supports network policies, such as Calico, allows you to restrict traffic from both external and internal networks.

Grant Access to Cloud DNS

Kubernetes Addons

source env.sh
helmfile --file
kube-addons/helmfile.yaml
source env.sh
helmfile --file
kube-addons/issuers.yaml
kubectl get all,certissuers --namespace kube-addons

Example Application: Dgraph

source env.sh
helmfile --file dgraph/helmfile.yaml
kubectl get ing --namespace dgraph
source env.sh
HTTP_ADDR=dgraph.${DNS_DOMAIN}
GRPC_ADDR=
grpc.$DNS_DOMAIN
GIT_ADDR=
raw.githubusercontent.com
GIT_PATH=dgraph-io/pydgraph/master/pydgraph/proto/api.proto
# test using HTTP/1.1
curl
$HTTP_ADDR/health | jq
curl $HTTP_ADDR/state | jq
# fetch api.proto file
curl -sOL https://$GIT_ADDR/$PATH
# test using gRPC
grpcurl -proto api.proto \
$GRPC_ADDR:443 \
api.Dgraph/CheckVersion

Example Application: Ratel

source env.sh
helmfile --file ratel/helmfile.yaml

Fun with Dgraph

Dataset and Schema through gRPC

~/projects/ingress-nginx-grpc
└── examples
└── pydgraph
├── Dockerfile
├── Makefile
├── helmfile.yaml
├── load_data.py
├── requirements.txt
├── sw.nquads.rdf
└── sw.schema

Query using Ratel

Troubleshooting

Certificates

source env.sh
curl -svvI https://dgraph.$DNS_DOMAIN/health

Cleanup

Kubernetes Resources

Google cloud resources

Resources

gRPC Articles

ingress-nginx

external-dns

cert-manager

Previous Article

Conclusion

Complexities of gRPC

Complexities of documentation

Continued Adventures of of gRPC

--

--

Linux NinjaPants Automation Engineering Mutant — exploring DevOps, o11y, k8s, progressive deployment (ci/cd), cloud native infra, infra as code

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Joaquín Menchaca (智裕)

Linux NinjaPants Automation Engineering Mutant — exploring DevOps, o11y, k8s, progressive deployment (ci/cd), cloud native infra, infra as code