Testing on the Chef Platform: Overview
Introduction to Test Automation with Chef
You awake from you deep slumber, trying to grasp fading memories of an amazing dream. You hear the sound of your phone alerts, pager duty, EDB (Enterprise Database) server not responding. Ouch.
Moments into troubleshooting, you see that the upgrade to EDB did not go as expected, and trace the problem to failing version of a cookbook included to a customer facing environment, where it shouldn’t. After some digging, you catch the problem, and fix it. It’s getting close 6 AM, and you’re wondering, can I get some sleep, or should I just go into work early…
…This may not be your experience, because we never had production outages, right? Am I right?
Overview
When developing infrastructure automation, especially in the area of change configuration, such as Chef cookbooks, one opportunity that is often neglected is writing some actual tests, despite the high risk that mistakes in infrastructure could cause.
With an application, you can deploy a new version of the software to fix an outstanding issue. With the infrastructure, typically you cannot easily deploy a new version, especially given all the complex inter-dependencies, not the mention existing applications running on the infrastructure.
For this reason alone, testing infrastructure before changing the infrastructure is vital.
Types of Testing
On the Chef platform, there are at three types of checks and testing supported: static analysis, unit testing and integration testing. This is an overview of the two types of testing, what they are, and why you might want to use them, or not use them.
Static Analysis
This is not really testing as far as executing code, but rather analyzing the code itself without running it. The process of performing static analysis will analyze the code and report back any common problems and style guide issues. This is usually a first step before doing any testing.
This initial analysis can inspect both ruby code usage and usage of the Chef interface.
Unit Test
In unit testing, we look at what is the minimal component that we can test, called the unit. In Chef, this would be a Chef recipe, or a custom resource. We look at this component, as an solitary or isolated unit, consider what is the the input and the output.
In Chef, the input for a recipe could come in the form of attributes, data bags, and search queries, and the input for a custom resource would be the properties. In reality thought, as a recipe is essentially Ruby code, the input can be just about anything you can code in Ruby. In unit testing, we would want to control these inputs, so that we can have deterministic results.
When will we run a recipe, the output would be the converged result of the convergence process: service started and enabled, configuration file updated to the desired state, packages installed, and so on. In a unit test run, these would be stubbed out and behaviors of what they should be will be observed.
Integration Testing
Integration tests determine if independently developed units of software work correctly when they are connected to each other. – Martin Fowler (Jan 2018)
Unlike unit testing in Chef, where the system is mocked and the inputs and outputs are stubbed out, in integration testing these are all real, thus integrated into a real system with a real chef run convergence into that system.
You can test a single cookbook and how that integrates into the system, with a narrow scope of testing a single cookbook, or create more robust tests that combine several cookbooks together. I would recommend both:
- Single Cookbook Integration: testing only a single cookbook, like Apache cookbook or a MySQL cookbook. The integration tests would live with the cookbook code.
- Multiple Cookbook Integration: testing several cookbooks together, such as a LAMP stack, which would use an Apache cookbook and MySQL cookbook on separate systems, and test the full integration of all the components working together. The integration test would exist outside the cookbook directory.
Systems Testing
Testing is conducted on a complete, integrated system to evaluate the system’s compliance with its specified requirements – IEEE 610.12–1990
The system in this case would be the whole infrastructure as a system, which includes all the cookbooks used to configure parts of the infrastructure, which depending on your environment, could also include parts that are configured with other automation or even manually hand crafted by humans, e.g. network equipment, NAS appliance, cloud resources, etc.
These are traditionally not tested, except through manual inspection initially, and later through alerts raised by monitoring, log aggregation, metrics, instrumentation, etc.
Testing solutions are starting arise with popularity of infrastructure as code, where automation goes beyond system resources (packages, services, configuration files) and allows you to programmatically create cloud resources (VPCs, gateways, subnets, load balancers, virtual instances, virtual storage).
The Testing Tools
CookStyle
Category: Static Analysis
Cookstyle is a linter tool that will analyze Ruby code and flag errors. It is based on RuboCop Ruby linting tool, but with style rules that make sense for cookbook development.
FoodCritic
Category: Static Analysis
FoodCritic is linter tool that evaluates Chef oriented ruby code for best practices, common mistakes, syntax, correctness, and style.
ChefSpec
Category: Unit Tests
ChefSpec is the framework for unit testing. The framework will test units, or “resources and recipes” in the context of Chef, “as part of a simulated chef-client run”.
The inputs to the units can be declared or stubbed out, and system attributes that are normally set by Ohai, are be mocked by Fauxhai, to determine the behavior of your cookbook on a mock system, such Ubuntu or CentOS.
TestKitchen
Category: Test Harness that enables Integration Testing
Kitchen provides a test harness to execute infrastructure code on one or more platforms in isolation. A driver plugin architecture is used to run code on various cloud providers and virtualization technologies such as Vagrant, Amazon EC2, and Docker – Test Kitchen website
With Test Kitchen you can launch systems, run your recipes (a process called convergence) on those same systems, and then verify that recipes work correctly using a test verifier.
The test verifier can be a solution like ServerSpec or InSpec, which are run locally on the system in question. This is the most optimal solution, with the highest value proposition.
- WebSite: https://kitchen.ci/
- Docs: https://docs.chef.io/kitchen.html
- GitHub: https://github.com/test-kitchen/test-kitchen
- ServerSpec: https://serverspec.org/
- InSpec: https://www.inspec.io/
InSpec
Category: Verifier used for integration testing, systems testing, and compliance
InSpec is a free and open-source framework for testing and auditing your applications and infrastructure – InSpec website
InSpec uses a transport system (from the Train library) that allows InSpec to execute remotely through SSH or WinRM, through docker exec, or locally. Train has recently added the ability to communicate through cloud providers like AWS, Azure, or Google Cloud, and recently VMWare.
Using InSpec to drive tests, you can test a whole system, potentially performing true systems testing, where you can test both the configured system resources (packages, services, files) and configured cloud resources (VPCs, subnets, load balancers, pub-sub, databases).
Should you have the need to test cloud resources, InSpec is currently the only option:
- WebSite: https://www.inspec.io/
- Docs: https://www.inspec.io/docs/
- GitHub: https://github.com/inspec/inspec
Practices and Trends
Currently, after ChefConf 18, there’s a recent emphasis on jumping straight into integration testing using Test Kitchen, and skipping unit testing altogether.
Some of this may have to do with potential low value return on unit tests, unless your code is complex with code logic or custom resources. Below are some practices gathered from Noah’s Dangers of Overtesting blog and further discussions, ChefSpec’s readme, and my own adventures…
Practices for Unit Tests
The documentation, training, and tutorials will encourage doing unit tests in the worst way possible, writing tautological tests to redundantly test the Chef itself, rather than test your code logic (if there is code logic).
Thus, you’d have a chef resource do this:
package 'apache2' { action: install }And an redundant test that test Chef:
it { expect(chef_run).to install_package 'apache2' } # <= Bad! Don’t do this, unless you have a lot of time to test on your hands to test Chef, rather than your code.
If you do write unit test, it makes sense to test different behaviors of code logic, testing a custom resource, and perhaps testing contents within a template for configuration file varies based on code logic (such as environments).
Practices for Integration Tests
In integration tests, the highest value you could have for tests the promise of what the cookbook should do, like install a web service with some content.
For example, with InSpec as the verifier, you could do this:
describe port(80) do
it { should be_listening }
enddescribe http('localhost') do
its('status') { should eq 200 }
its('headers.Content-Type') { should include 'text/html' }
its('body') { should include 'Hello World!' }
end
Low value tests would test the implementation rather than the promise, which in turn test Chef again, to make sure Chef did what it was suppose to do on a system.
Further, this style could have negative value if you support more than one operating system, because you have to test variances for each system, such as a package called apache2 or apache or httpd or www-servers/apache. This leads to opportunity to introduce bugs in the tests, and the need to test the tests that test tests. Just don’t do it…
