DevOps Tools: Introducing Helmfile

Automate Helm Charts with Helmfile

In the Kubernetes community, it would be a surprise to find anyone that does not yet know about the popular Helm tool to deploy services. Similar to tools like Homebrew for macOS or Chocolatey for Windows, you can install a solution on Kubernetes easily with .

Using helm charts

Helm charts share one thing in common with change config tools like Chef, Consul-Template, Ansible, Puppet, or Salt Stack, where you can use a template engine like ERB or Jinja to dynamically compose a configuration file. With Helm, the configuration files are Kubernetes manifests, which are dynamically built using go-template template engine with the Sprig library and customized with with values you pass during deployment.

As an example, you can enable an endpoint and storage in a typical chart like this:

The tool also supports passing in a configuration file that contains these values, so you can achieve the same results with the following:

Introducing Helmfile

A Helm chart package is useful for installing a single service on Kubernetes. Here’s the problem: What happens when you need to install several charts with values that need to be coordinated with all of the charts?

This is where the tool like helmfile is useful. With a single configuration script (), you can install multiple charts and supply a set of values for all of the charts.

Using the example from above, we can compose same thing using as the following with the Helm chart values embedded into .

If you wanted the values in a separate external values file, you can do the same thing with the following:

A Complete Example

This is an full example using to integrate two solutions, a highly performant distributed graph database called Dgraph with an object storage (buckets) solution using MinIO.

The Helmfile Script

Below is an example for MinIO and Dgraph helm charts. When using this configuration, helmfile will add the Helm chart repositories and then apply these two Helm charts with values files found in directory.

helmfile.yaml

MinIO Chart Values

Below is a MinIO configuration that references a recent image of MinIO and supplies access key and secret key, as well as a default bucket called . This bucket is created using the command line tool from a batch job that is run during deployment.

values/minio.yaml.gotmpl

Dgraph Chart Values

Below is a Dgraph configuration that configures backups to be sent to a MinIO server. The Dgraph helm chart will configure a Kubernetes Cronjob that runs a script to instruct Dgraph to do backups to this remote MinIO server.

values/dgraph.yaml.gotmpl

The script that supports backups will issue RESTful or GraphQL request to Dgraph with appropriate support for Access JWT tokens (see ACLs) and Mutual TLS (see TLS) if these features are enabled. This script works with Dgraph v21.03.0 and earlier versions, using GraphQL if supported in the Dgraph version.

Applying this Configuration

Before we begin, in addition to downloading the above code snippets, we need to set some environment variables that can be used to inject values into our helmfile configuration.

First, download the above code snippets and create a directory structure like the following:

You can set these required and optional environment variables:

Once these are set, you can just run the following:

After this, if MinIO and Dgraph are installed into the same namespace, you should something similar to the following resources with :

Example Output on RKE cluster

Testing the Results

The MinIO has a graphical interface, which you can view using port-forward:

After running this command, you can log in to MinIO at http://localhost:9000:

After you log in, you should see a single bucket :

After about 15 minutes, which is the time set for full backups in the configuration (, you should see the contents created by Dgraph:

Should there be an error with the backups, you can check the status using the following method:

NOTE (Troubleshooting): Starting in Dgraph v21.03.0, errors will not be reported from a failed backup request. In previous versions of Dgraph, backups were synchronous (blocking), so if the an error occurred, such as bad MinIO address, credentials, licensing, etc., you can get an error back immediately.

However in Dgraph v21.03.0, the backups are now asynchronous (non-blocking running the background) so you will have to consult the logs and guess which log entry is connected to the triggered backup request based on the timestamp.

For example:

TIP (Alerting): This is out of the scope of this simple example, but should you explore setting up some observability to capture backup failures, this is how you can go about doing this.

Previous to Dgraph v21.03.0, you can setup alerts using something like kube-state-metrics and scan for Kubernetes Cronjob failures related to backup cron job labels. As an example using Prometheus AlertManager, here’s an example: dgraph-backup-alert-rules.yaml.

With Dgraph v21.03.0, you will need configuring log heuristics to detect and report errors from backup events.

Resources

Here are some links that I have come across on the topics in this article:

Helm

Helmfile

Text Editors

Color syntax highlighting of gotemplate yaml files:

Helm charts

Source code for this tutorial

Conclusion

As you can see, Helm charts when combined with helmfile is quite powerful. Using these together, you can deploy an entire organizational infrastructure on Kubernetes within minutes.

This article would not be complete without mentioning that there are alternatives to Helm and helmfile

Kustomize as an alternative

Another tool that fills the same role as Helm charts is kustomize, which is built into the Kubernetes client tool () or available externally as command. This does not use templates or variables, but directories of manifests and kustomize files that are used to patch Kubernetes manifests during deployment.

Personally, I find this harder to manage a directory of patch files for complex solutions. In practice, errors are quite common, and the solution becomes both ungainly and unmanageable quite fast. Still, if Kubernetes operators are used and there are no Helm charts to built around the operator, then kustomize may be the only solution.

Pros

  • Standardized Tooling: the kustomize tooling is embedded into the command, so no further tooling is required.
  • Helm is Unavailable: some solutions such as Kubernetes operators may not have a Helm chart required to dynamically create manifests, so kustomize is an available alternative

Cons:

  • Complexity: Managing directories of patch files can be difficult to troubleshoot and debug.
  • Maintenance: Managing directory of patches can get ungainly and unmanageable quickly, and with numerous small files, it is error prone.

If you would like to integrate kustomize solutions into helmfile, you’ll be happy to know that there are two methods to support kustomize. inside your helmfile scripts:

  • run an external script like when deploying a chart, where this script can use kustomize to dynamically compose a Helm chart. (Thanks to Yusuke Kuoka for this solution)
  • support kustomize syntax directly inside helmfile. This may require a helm plugin, and is currently not documented except in the form of go code. (Thanks to Yusuke Kuoka for making this happen)

A big shout out to Yusuke Kuoka for kustomize + helmfile integration and so many wonderful contributions to helmfile core tool. Forever thank you.

Terraform as an alternative

Another solution is to use Terrafom to manage Kubernetes resources through the Kubernetes provider and the Helm provider, in combination with an embedded template engine using the HCL language.

Using this solution, you could provision cloud resources, and based on those result of these resources created (such as uids), deploy Helm charts using values based on the cloud resources. For example: create a GKE cluster and a GCS bucket, then deploy charts with values that reference the GCS bucket on the same GKE cluster. The Kubernetes credentials required for access would come from the provisioned GKE cluster.

Pros:

Cons:

  • Integration (Segregated Provisioning): Integration between resources managed by different providers is not completely seamless due to how Terrafom evaluates provider blocks verses the actual resources (see Provider Configuration). Essentially resources should not be created in the same Terrafom module where Kubernetes provider resources are also utilized.
  • Complexity (Different Tools): Though you may share the same tooling in Terrafom language (HCL) for cloud and Kubernetes resources, Terrafom is not as popular as pure and tools used by most of the community, and Terraform’s template engine used by templatefile is not popular, at least compared to gotemplate.
  • Complexity (Source of Truth): The source of truth are not the live Kubernetes resources themselves as reported by the Kubernetes API, but rather an offline state file that needs to be synchronized with the actual online state. This has been more of a hassle for cloud resources than Kubernetes provider resources in practice, so this issue may be negligible.
  • Maintenance (Dependency Hell): There can be dependency nightmares where changes to versions of the command line tool, providers, modules, as well as module dependencies, can cause disruptive changes, where infrastructure must be destroyed and recreated.
  • Quality or Feature Limitations: Terrafom providers that allow you to interact with cloud resources have had and will have bugs that will not exist in the Kubernetes API accessed by or Helm. For features, such as using kubeconfig files, this is not supported (or had problems) in later versions of the Kubernetes provider. Some functions like jsonencode will convert booleans and ints into strings, creating undesirable JSON types. Many gotcha and poignant issues like this are automatically closed through Hashicorp installed git bot automation, as no one from Hashicorp has given the issue proper attention.

The above may seem like I do not like Terrafom, but honestly, I love both Terrafom and helmfile, and think both are great solutions. If you would like to integrate both helmfile and Terrafom together, there’s a helmfile provider that you can explore.

A big shout out to Yusuke Kuoka for helmfile provider

Change Configuration as an alternative

Another solution is to actually use a change configuration platform like Puppet or Chef, as these use a template engine that can dynamically compose Kubernetes manifests, or may have modules or cookbooks that talk directly to the Kubernetes API.

Pros

  • Standardized Platform: Use the same tooling for both Kubernetes resources with immutable infrastructure as well as traditional configuration management with mutable infrastructure.

Cons

  • Unnecessary (immutable images): Many of the features around a change configuration platform like Puppet or Chef to manage mutable infrastructure, where configuration state is converged to the desired state will not be used given that container images are immutable.
  • Redundant (state already managed): A change configuration platform like Puppet or Chef is redundant, as the infrastructure components deployed by Kubernetes are managed by an embedded reconciliation loop, similar to a convergence loop to manage mutable infrastructure.
  • Complexity: Change configuration platforms like Puppet or Chef are complex and domain specific (have their own DSLs or interface), so this adds further unnecessary complexity, especially when most of the features around convergence will never be used.
  • Expensive: given the above complexity, there be added costs in time and resources, in both lead time, testing, and maintenance. Furthermore, these platforms can have excessive licensing costs depending on usage.

The discussion above is oriented around pull-based agent solutions (Chef, Puppet, CFEngine), so some of this may not be applicable to a push-based agent-less tool like Ansible or other remote-execution platforms like Salt Stack.

Ansible in some ways is similar to Terrafom in regards to provisioning of cloud resources, except that Ansible doesn’t maintain an offline state file, but uses the source of truth from direct interaction with Kubernetes or other cloud web API.

Wrapping up

I find helmfile to be the most flexible and efficient tool to manage Helm charts, and the Helm chart itself to be the most effective way to manage dynamic Kubernetes manifests in practice.

There may be some niche situations where you may want to use kustomize or Terrafom. As for change configuration like Puppet or Chef, I would never recommend this as it adds an incredible level of complexity and expense (lead time, maintenance, and licensing). Just don’t do it.

Linux NinjaPants Automation Engineering Mutant — exploring DevOps, o11y, k8s, progressive deployment (ci/cd), cloud native infra, infra as code

Linux NinjaPants Automation Engineering Mutant — exploring DevOps, o11y, k8s, progressive deployment (ci/cd), cloud native infra, infra as code