Open in app

Sign in

Write

Sign in

Home

Library

Stories

Stats

TestKitchen with Chef

Joaquín Menchaca (智裕)
8 min readAug 29, 2018

--

Test Kitchen is a remarkable tool that not only creates and provisions systems, but also tests them to verify that your scripts correctly configured the systems.

This tool fits into the category of a test harness, where the program unit are the configuration scripts, such as Chef recipes, and the test execution engine would involve tools like ServerSpec or InSpec to run tests on the system.

This small tutorial is a rudimentary overview of Test Kitchen using ServerSpec (through the default Busser layer). I also added a section on using alternative verifiers like the kitchen-inspec plugin.

Previous Guide

In a previous guide, I demonstrated how to use Vagrant to bring up a system and provision it with Chef.

Vagrant Provisioning with Chef

Provisioning Systems with Vagrant and Chef-Zero

medium.com

This guide is essentially the same thing, provisioning a system through Chef, from Test Kitchen instead of Vagrant., and then an additional step of testing a system through either Busser with ServerSpec or an external verifier like the popular InSpec.

Prerequisites

This tutorial requires the following tools:

Additionally, this guide is written with instructions for Bash. If you use an another shell, like Zsh or PowerShell, you’ll need to convert the instructions that do the same thing.

Previous Guides

I have written some previous guides that show how to get these tools for macOS, Windows, and Linux.

Windows 8.1 using Chocolatey:

VirtualBox and Friends on Windows 8.1

VirtualBox, Vagrant, Test Kitchen, Docker Machine, Minikube

medium.com

macOS High Sierra 10.3 using Homebrew:

VirtualBox and Friends on macOS

VirtualBox, Vagrant, Test Kitchen, Docker Machine, Minikube

medium.com

Fedora 28:

Virtualbox and Friends on Fedora 28

VirtualBox, Vagrant, Test Kitchen, Docker Machine, Minikube

medium.com

Installing using Ruby (Advanced)

You can install it manually using Ruby 2.5.1 or later. I highly recommend using a Ruby manager like rbenv or rvm. With Bundler, you can get recent Test Kitchen with this Gemfile:

source :rubygemsgem 'test-kitchen'
gem 'kitchen-vagrant'
gem 'kitchen-inspec'
gem 'inspec'
gem 'berkshelf'

Part I: Create Chef Cookbook

This is the same process as the Vagrant guide, which will install create a minimal Chef cookbook structure, modeled after the chef generate cookbook command in Chef 14.

Create Cookbook

PROJ_PATH=${HOME}/vagrant-chef
COOKBOOK_PATH=${PROJ_PATH}/cookbooks/hello_web
mkdir -p ${COOKBOOK_PATH}/{files/default,attributes,recipes}cd ${COOKBOOK_PATH}##### Create Cookbook Attributes
cat <<-'ATTRIB' > attributes/default.rb
default['hello_web']['package'] = 'apache2'
default['hello_web']['service'] = 'apache2'
default['hello_web']['docroot'] = '/var/www/html'
ATTRIB##### Create Cookbook Recipe
cat <<-'RECIPE' > recipes/default.rb
apt_update 'Update the apt cache daily' do
  frequency 86_400
  action :periodic
endpackage node['hello_web']['package']cookbook_file "#{node['hello_web']['docroot']}/index.html" do
  source 'index.html'
  action :create
endservice node['hello_web']['service'] do
  supports status: true, restart: true, reload: true
  action %i(enable start)
end
RECIPE##### Create Cookbook Metadata
cat <<-'METADATA' > metadata.rb
name 'hello_web'
version '0.0.1'
chef_version '>= 12.14' if respond_to?(:chef_version)
METADATA##### Create Content
cat <<-'HTML' > files/default/index.html
<html>
  <body>
    <h1>Hello World!</h1>
  </body>
</html>
HTML

When finished, this will create a structure that looks like this:

.
├── attributes
│   └── default.rb
├── files
│   └── default
│       └── index.html
├── metadata.rb
└── recipes
    └── default.rb

Part II: Configure Test Kitchen

In this part, we will configure our test structure and tests needed to verify the cookbook we created in the previous section.

Initial Test Structure

We can create the Test Kitchen structure within our cookbook with the following below:

PROJ_PATH=${HOME}/vagrant-chef
COOKBOOK_PATH=${PROJ_PATH}/cookbooks/hello_web
SERVERSPEC_PATH=${COOKBOOK_PATH}/test/integration/default/serverspeccd ${COOKBOOK_PATH}mkdir -p ${SERVERSPEC_PATH}
touch .kitchen.yml \
  ${SERVERSPEC_PATH}/{default_spec.rb,spec_helper.rb}

This should result in structure looking like this with additions in bold:

.
├── .kitchen.yml
├── attributes
│   └── default.rb
├── files
│   └── default
│       └── index.html
├── metadata.rb
├── recipes
│   └── default.rb
└── test
    └── integration
        └── default
            └── serverspec
                ├── default_spec.rb
                └── spec_helper.rb

Configure Test Harness

Now it is time configure Test Kitchen to use Chef and run a cookbook on Ubuntu platform.

---
driver:
  name: vagrantprovisioner:
  name: chef_zero
  always_update_cookbooks: trueplatforms:
  - name: ubuntu-16.04suites:
  - name: default
    run_list:
      - recipe[hello_web::default]

The chef_zero provisioner will look for the key run_list to determine what should run on the node.

Create A Test

Let’s update our default spec test default_spec.rb:

require 'spec_helper'describe port(80) do
  it { should be_listening }
end

We also want to update our spec helper spec_helper.rb:

require 'serverspec'set :backend, :exec

This is necessary to add this small part, or we’ll get spammed with this message six times for each run:

No backend type is specified. Fall back to :exec type.

These messages will disappear and ServerSpec will be instructed to use the local execution for the backend.

Part III: Testing a Single Platform

Creating Systems

To get started, we can create the Ubuntu system:

kitchen create

If you have not downloaded the Vagrant box images, i.e. vagrant box add bento/name_of_system, it will be downloaded now, and then the system(s) will be brought up.

Check Working System

To verify things are working, run:

kitchen exec ubuntu -c 'lsb_release -a'

This should show us something like:

-----> Execute command on default-ubuntu-1604.
       No LSB modules are available.
       Distributor ID: Ubuntu
       Description: Ubuntu 16.04.4 LTS
       Release: 16.04
       Codename: xenial

Configure the System

Now with at least one working system, let’s download chef client and configure it with:

kitchen converge

This should configure the system with the hello_web cookbook.

Note: If for some reason the download of Chef Client failed, Test Kitchen will save the bad download for installing Chef on other systems, and will give a stack trace as feedback. To fix this, you will need to purge the bad downloaded packages (such as chef-14.3.37–1.el7.x86_64.rpm or chef_14.3.37–1_amd64.deb) that are found in ~/.kitchen/cache/.

Test the System

Now we can run the tests:

kitchen verify

This should show something like the following:

-----> Starting Kitchen (v1.22.0)
-----> Verifying <default-ubuntu-1604>...
       Preparing files for transfer
-----> Busser installation detected (busser)
-----> Busser plugin detected: busser-serverspec
       Removing /tmp/verifier/suites/serverspec
       Transferring files to <default-ubuntu-1604>
-----> Running serverspec test suite
       /opt/chef/embedded/bin/ruby -I/tmp/verifier/suites/serverspec -I/tmp/verifier/gems/gems/rspec-support-3.8.0/lib:/tmp/verifier/gems/gems/rspec-core-3.8.0/lib /opt/chef/embedded/bin/rspec --pattern /tmp/verifier/suites/serverspec/\*\*/\*_spec.rb --color --format documentation --default-path /tmp/verifier/suites/serverspecPort "80"
         should be listeningFinished in 0.1038 seconds (files took 0.29169 seconds to load)
       1 example, 0 failuresFinished verifying <default-ubuntu-1604> (0m1.47s).
-----> Kitchen is finished. (0m1.90s)

Part IV: Testing Multiple Platforms

Now let’s have some fun and add more platforms besides just Ubuntu. We can throw in CentOS and FreeBSD.

Adding Platforms

Update Test Kitchen configuration (.kitchen.yml) to the following:

---
driver:
  name: vagrantprovisioner:
  name: chef_zero
  always_update_cookbooks: trueplatforms:
  - name: ubuntu-16.04
  - name: centos-7
  - name: freebsd-11.2suites:
  - name: default
    run_list:
      - recipe[hello_web::default]

Try the new configuration out:

kitchen create
kitchen converge

This should fail because your cookbook only supports Ubuntu. You will spot errors like these:

FATAL: Chef::Exceptions::Package: yum_package[apache2] (hello_web::default line 6) had an error: Chef::Exceptions::Package: No candidate version available for apache2FATAL: Mixlib::ShellOut::ShellCommandFailed: freebsd_package[apache2] (hello_web::default line 6) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '69'

The package name for Apache httpd server is different across the platforms. To support alternative package name, alone with the service name, and docroot, we need to override the default values on each platform.

Overriding Attributes

Update the Test Kitchen configuration (.kitchen.yml) to look like the following:

---
driver:
  name: vagrantprovisioner:
  name: chef_zero
  always_update_cookbooks: trueplatforms:
  - name: ubuntu-16.04
  - name: centos-7
    attributes:
      hello_web:
        package: httpd
        service: httpd
        docroot: /var/www/html
  - name: freebsd-11.2
    attributes:
      hello_web:
        package: apache24
        service: apache24
        docroot: /usr/local/www/apache24/data/suites:
  - name: default
    run_list:
      - recipe[hello_web::default]

Run kitchen converge to see the results. We should spot lines that say yum_package[httpd] action install for CentOS and freebsd_package[apache24] action install for FreeBSD.

Test Multiple Platforms

Run kitchen verify to see the results. On all systems, we should see this flash by for each platform:

Port "80"
         should be listening

Part V: Busser vs. InSpec

In Test Kitchen, the default verifier uses Busser. Specifically, Test Kitchen will attempt to install a Busser runner plug-in gem based on the directory path name. So if you have test/integration/$SUITE/$VERIFIER, Test Kitchen will do a gem install busser-$VERIFIER, where $VERIFIER is the test tool like testinfra, goss, pester, bats, or serverspec.

There are alternative verifiers that do not transit through the Busser layer, and follow their own particular configuration schema for verification. One of these, or the most popular verifier, is InSpec with the kitchen-inspec plugin.

You can use InSpec instead of the default Busser by adding this to your Test Kitchen configuration (.kitchen.yml):

verifier:
  name: inspec

After this you can to put your tests in the default path of test/integration/$SUITE/. In this tutorial you can add a default test file with:

PROJ_PATH=${HOME}/vagrant-chef
COOKBOOK_PATH=${PROJ_PATH}/cookbooks/hello_web
INSPEC_PATH=${COOKBOOK_PATH}/test/integration/default/cd ${COOKBOOK_PATH}cat <<-'INSPEC' > ${INSPEC_PATH}/default_test.rb
describe port(80) do
  it { should be_listening }
end
INSPEC

The full configuration (.kitchen.yml) with the InSpec verifier will look like:

---
driver:
  name: vagrantprovisioner:
  name: chef_zero
  always_update_cookbooks: trueverifier:
  name: inspecplatforms:
  - name: ubuntu-16.04
  - name: centos-7
    attributes:
      hello_web:
        package: httpd
        service: httpd
        docroot: /var/www/html
  - name: freebsd-11.2
    attributes:
      hello_web:
        package: apache24
        service: apache24
        docroot: /usr/local/www/apache24/data/suites:
  - name: default
    run_list:
      - recipe[hello_web::default]

After this, try it out:

kitchen verify

You’ll get similar results with ServerSpec with reports that look like this:

Port 80
     ✔  should be listening

There will be an exception for FreeBSD, which does not yet have full support from InSpec.

Port 80
     ↺  The `port` resource is not supported on your OS yet.

Additional Busser Notes

I wanted to document Busser because there was not much in the way of documentation, and there are quite a few Busser plug-ins out there now. Currently though, in the Chef community, InSpec is the preferred verifier and not without good reasons.

In the scope of Test Kitchen alone, a Busser system will treat the system as a isolated system and install the verifier locally, then proceed to test executing them locally. This adds an additionally time tax and in some cases, can situation where it is hard to determine if there is a test bug (see article Test Kitchen の Shell Verifier で Serverspec による Cookbook テストを行う), or issue with the test in the Busser environment according to. Contrast to InSpec which uses ssh to remotely test Linux systems in this tutorial.

In the scope of InSpec compared to other test systems, such as ServerSpec, there are some good points why InSpec is preferred. I attempted a comparison earlier from the perspective of using either InSpec or ServerSpec to drive testing:

ServerSpec vs. InSpec

When discussing my article on ServerSpec in the scope integration testing, a colleague of mine asked me why not use…

medium.com

Currently, the Busser does not receive much love today (meaning focus or enthusiasm for updating or supporting it is not there). Fortunately, at least for ServerSpec, you can use it outside of Busser, with kitchen-verifier-serverspec. There are currently some great Busser plug-ins that I hope can be ported to direct verifiers in the future, until then, you’ll have to use the shell verifier.

I hope this is useful to show where things are at currently, as well and the background of Test Kitchen, Busser-based verifiers, and other Kitchen verifiers.

Conclusion

I hope this was both useful and fun. The take aways from this are:

  • Basic usage of Test Kitchen to configure multiple systems with Chef, and test multiple systems.
  • Override default attributes with alternative attributes per platform.
  • Verification on using ServerSpec with default Busser or the InSpec verifier.
  • Some differences and trade-offs between Busser plug-ins and direct Kitchen verifier plug-ins.
Chef
Serverspec
Testkitchen
DevOps
Inspec

--

--

Joaquín Menchaca (智裕)
Joaquín Menchaca (智裕)

Written by Joaquín Menchaca (智裕)

1.6K Followers
139 Following

DevOps/SRE/PlatformEng — k8s, o11y, vault, terraform, ansible

No responses yet

Write a response

What are your thoughts?

Help

Status

About

Careers

Press

Blog

Privacy

Rules

Terms

Text to speech