Securing GKE with Managed SSL

Using Google Managed SSL Certificates with a GKE Cluster

6 min readAug 2, 2020

There comes a time where you will need to secure web traffic for your web services on Kubernetes. Alright, that’s all the time. For GKE, you can do this with Google Managed SSL certificates.

Certificate management requires having a registered domain name. Thus, for this article, you will need to have a registered domain or subdomain on Cloud DNS and a GKE cluster with ability to update DNS records using External DNS.

This article will show walk you through using Google Managed SSL certificates with GKE.

Prerequisites

Tool Requirements

Google Cloud SDK that is authorized to your google account and your account has access to a Google Project where you can manage GCE, GKE, and Cloud DNS.
Kubectl (pronounced koob-cuttle) is the Kubernetes client command line tool to interaction with the cluster and installing Kubernetes manifests.

Infrastructure Requirements

Registered Domain or Sub-Domain on Cloud DNS.
GKE Cluster (Kubernetes 1.16 or higher) configured scope to access Cloud DNS API.
Installation of ExternalDNS on GKE Cluster configured with access to update records on Cloud DNS.

In a previous article, I demonstrated how to configure ExternalDNS to update cloud DNS records (GKE and Cloud DNS must be in the same GCP project):

Extending GKE with ExternalDNS

Extending Kubernetes to update Cloud DNS records from GKE

medium.com

Deploy Application with SSL Support

We will need to use an Ingress resource to create our endpoint for L7 HTTP/S support. Below is a diagram the resources used in this exercise:

By now you should be familiar with Deployment controller to managed Pods and creating a Service resource to balance traffic to the Pods.

We will create a certificate using Google’s CRD (custom resource definition) called ManagedCertificate to provision a new SSL certificate.

To connect all of this together, we create an Ingress resource (using the default ingress class of gce) that does the following:

provision an external L7 load balancer on Google Cloud
register a new DNS record that points to the external IP address of the load balancer (using ExternalDNS)
attach certificate that we declared earlier to the load balancer, which also completes the certificate provisioning process.

Before starting this, you should have created GKE cluster with ExternalDNS installed an configured, as mentioned above.

Now let’s begin…

Deploy the Deployment Controller

First we can deploy a deployment to manage an create the pods hosting the web application. Create a file gce_ssl_deploy.yaml with the following contents:

Deploy this with:

kubectl create --filename gce_ssl_deploy.yaml

Deploy the ManagedCertificate CRD

Managing external load balancers on Google Cloud can be complex, as you’ll need to create forwarding rules, target proxy rules, register backend services, and the certificate, which is added to a HTTP target proxy.

Fortunately, this is created indirectly using an Ingress annotation, and managed certificate CRD (Custom Resource Definition). Create a file named gce_ssl_managed_cert.yaml with the following content.

Replace $MY_DNS_NAME with a domain name you want to register.

If you have bash shell and GNU Sed, you can do this with this command:

MY_DNS_NAME=<put_your_domain_here> # hello.test.acme.comsed -i "s/\$MY_DNS_NAME/$MY_DNS_NAME/" gce_ssl_managed_cert.yaml

Deploy this edited manifest:

kubectl create --filename gce_ssl_managed_cert.yaml

Deploy the Service

We need to deploy the service resource with NodePort type (ClusterIP will not work with gce ingress). Create a file gce_ssl_service.yaml.

Deploy this manifest:

kubectl create --filename gce_ssl_service.yaml

Deploy the Ingress Resource

Now comes time for the final piece the ingress resources, which will read our desired certificate as well create an external load balancer on Google Cloud. Additionally, this will register a DNS record on Cloud DNS using external-dns (which should be setup already).

Create a file gce_ssl_ingress.yaml with the following content.

Replace $MY_DNS_NAME with a domain name you want to register. If you have bash shell and GNU Sed, you can do this with this command:

MY_DNS_NAME=<put_your_domain_here> # hello.test.acme.comsed -i "s/\$MY_DNS_NAME/$MY_DNS_NAME/" gce_ssl_ingress.yaml

Deploy this edited manifest:

kubectl create --filename gce_ssl_ingress.yaml

Verify Certificate is Available

This process can take a while from 10 to 20 minutes for the certificate to move from a Provisioning to an Active status.

Check with kubectl

You can verify the state of the certificate in Kubernetes with the following command:

kubectl describe \
  managedcertificates.networking.gke.io/hello-k8s-gce-ssl

The results will look something like this when it becomes Active:

Check with gcloud

You can also check with gcloud as well:

# SAN cert can have multiple domains, so newline as separator
DOMAINS_COL="managed.domains[].list(separator="$'\n'")"
# Format string with multi-line domain column
FORMAT="table[box](name,type,managed.status,$DOMAINS_COL)"gcloud compute ssl-certificates list \
 --filter "hello." \
 --format "$FORMAT"

The output should look something like this

Test the Web Application

Once finished, you can use HTTPS to access the service, such as our example above https://hello.test.acme.com.

Clean-up

You can remove the deployed resources with the following command:

cat hello_*.yaml | kubectl delete --filename -

Notes on Google Managed SSL Certificates

The automation to create trusted certificates is great, but there are some things to consider then crafting your certificates:

Provisioning an Ingress, Certificate, and DNS can take significant time, so using reserved static external IP address is recommended.
no Wildcard certificates are supported, so you cannot register a certificate for *.test.acme.com for example.
SAN (Subject Alternative Name) certificate can be used with Kubernetes 1.16 and above to allow multiple address, such as web1.test.acme.com, web2.test.acme.com, web3.test.acme.com, and so on.
Every domain specific in the SAN certificate must be provisioned (ingress resource to create external load balancer and corresponding DNS record in Cloud DNS) or the certificate provisioning for all domains will never complete and certificates will not be available to secure traffic.

Managed SAN Certificate Behavior

On the requirement where domains specified in the SAN certificate must be used, this is not explicitly documented (at least in what I have researched) but I observed this behavior in y testing. Until all the certificated are attached to the external load balancer, they will remain in the status of PROVISIONING.

The domains that are used with a provisioned load balancer through the ingress resource will have a status of Active, while the domains that are not used with a provisioned load balancer will have a state of FailedNotVisible.

All domains specified in the SAN certificate will not have functional HTTPS traffic until they are used with a provisioned load balancer.

Reserved External IP Address

Deleting and recreating ingress resource is both disruptive and a time consuming process as a new ephemeral IP address will be created and DNS records pointing to the new address will have to be updated. For this reason, a reserved static external IP address is recommended.

You can create an external reserved IP address with gcloud command:

ADDRESS_NAME=acme-address-name# create address
gcloud compute addresses create $ADDRESS_NAME --global# retrieve the IP address
gcloud compute addresses describe $ADDRESS_NAME --global | \
  awk '/^address:/{print $2}'

You can add this annotation to your ingress resource to use this IP address:

kubernetes.io/ingress.global-static-ip-name: acme-address-name

Resources

Blog Source Code

Source Code: https://github.com/darkn3rd/blog_tutorials/tree/master/kubernetes/gke_5_googlessl

Google Documentation

Google-Managed SSL Certificates in GKE: https://cloud.google.com/kubernetes-engine/docs/how-to/managed-certs
Google-Managed SSL Certificates in GCP with LoadBalancer: https://cloud.google.com/load-balancing/docs/ssl-certificates/google-managed-certs
Reserved Static External IP Address: https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address

Conclusion

This article is my fifth article in my new GKE series that shows you how to generally manage a stateless web applications with GKE flavor of Kubernetes.

And this is just the beginning of the journey, as a robust infrastructure would include things like stateful backend applications, secure management of services (VPN and bastion jump host), log aggregation, monitoring with metrics, tracing and instrumentation, as well as more advance scenarios with application meshes.

I hope this helps with your Kubernetes and GKE adventures.