Testing Chef Cookbooks with InSpec

Integration Testing using Test Kitchen and Inspec

8 min readSep 1, 2018

For integration testing of systems with Chef, you can use the tool InSpec, either directly to drive tests, or use the test harness Test Kitchen. The advantage of using the test harness is that it can setup the systems, configure them, and then test them with InSpec. With InSpec alone, so you would have to create the systems and configure them beforehand.

This tutorial will walk you through using InSpec with Test Kitchen to test a Chef cookbook on one or more platforms.

Prerequisites

For integration testing journey with Chef, you will need the following prerequisites:

ChefDK — bundles Test Kitchen, InSpec, Chef as well as other tools needed for testing Chef.
Vagrant — automation tool used by Test Kitchen to manage virtual guest systems.
Virtualbox — free open-source virtual management tool for macOS, Windows, and Linux that is used by default with Vagrant.

Additionally, the instructions are written for Bash 4, so you may need to convert the instructions to your preferred shell, like Zsh or PowerShell, or install Bash.

Previous Guides

I have written some previous guides that show how to get these prerequisites for macOS, Windows, and Linux.

Windows 8.1 using Chocolatey:

VirtualBox and Friends on Windows 8.1

VirtualBox, Vagrant, Test Kitchen, Docker Machine, Minikube

medium.com

macOS High Sierra 10.3 using Homebrew:

VirtualBox and Friends on macOS

VirtualBox, Vagrant, Test Kitchen, Docker Machine, Minikube

medium.com

Fedora 28:

Virtualbox and Friends on Fedora 28

VirtualBox, Vagrant, Test Kitchen, Docker Machine, Minikube

medium.com

Install Test Kitchen and InSpec with Ruby (Advanced)

As an alternative to installing ChefDK, for more advanced scenarios where you may manage your own Ruby environment, you can install the required tools with the Gemfile below.

source :rubygemsgem 'test-kitchen'
gem 'kitchen-vagrant'
gem 'kitchen-inspec'
gem 'inspec'

I have made previous guides about rvm and rbenv for managing Ruby environments should you wish to explore this:

A Tale of Two Rubies, Part I

Managing Rubies with RVM

medium.com

and

A Tale of Two Rubies, Part II

Managing Rubies with Rbenv

medium.com

And a follow on article should you want to use one of these Ruby managers and ChefDK together:

Three Rubies and a ChefDK

ChefDK requires that you use their embedded ruby for running the Chef tools, but you may already have an existing ruby…

medium.com

Part I: Getting Started

The instructions in this part will create small project area and our basic Chef cookbook.

This is similar to other tutorials I have written using hello_web cookbook, but this one in particular uses dynamic attributes, where the attributes are set based on the platform family.

Creating Chef Cookbook

Create the basic structure and the populate the cookbook:

PROJ_PATH=${HOME}/vagrant-chef
COOKBOOK_PATH=${PROJ_PATH}/cookbooks/hello_web
mkdir -p ${COOKBOOK_PATH}/{files/default,attributes,recipes}
cd ${COOKBOOK_PATH}##### Create Cookbook Attributes
cat <<-'ATTRIB' > attributes/default.rb
# Dynamic Attributes based on Platform Family
case node['platform_family']
when 'rhel'
  default['hello_web']['package'] = 'httpd'
  default['hello_web']['service'] = 'httpd'
  default['hello_web']['docroot'] = '/var/www/html'
when 'debian'
  default['hello_web']['package'] = 'apache2'
  default['hello_web']['service'] = 'apache2'
  default['hello_web']['docroot'] = '/var/www/html'
when 'freebsd'
  default['hello_web']['package'] = 'apache24'
  default['hello_web']['service'] = 'apache24'
  default['hello_web']['docroot'] = '/usr/local/www/apache24/data/'
else
  default['hello_web']['package'] = nil
  default['hello_web']['docroot'] = nil
  default['hello_web']['service'] = nil
end
ATTRIB##### Create Cookbook Recipe
cat <<-'RECIPE' > recipes/default.rb
apt_update 'Update the apt cache daily' do
  frequency 86_400
  action :periodic
endpackage node['hello_web']['package']cookbook_file "#{node['hello_web']['docroot']}/index.html" do
  source 'index.html'
  action :create
endservice node['hello_web']['service'] do
  supports status: true, restart: true, reload: true
  action %i(enable start)
end
RECIPE##### Create Cookbook Metadata
cat <<-'METADATA' > metadata.rb
name 'hello_web'
version '0.0.2'
chef_version '>= 12.14' if respond_to?(:chef_version)
METADATA##### Create Content
cat <<-'HTML' > files/default/index.html
<html>
  <body>
    <h1>Hello World!</h1>
  </body>
</html>
HTML

This will lay out the similar structure that is created with a chef generate cookbook hello_web using Chef 14. The cookbook will install, start, and enable Apache HTTPD server and place our Hello World into the default document root.

The final result should look like:

~/vagrant-chef
└── cookbooks
    └── hello_web
        ├── attributes
        │   └── default.rb
        ├── files
        │   └── default
        │       └── index.html
        ├── metadata.rb
        └── recipes
            └── default.rb

Creating Test Testing Structure

Create the basic structure for Test Kitchen and InSpec:

PROJ_PATH=${HOME}/vagrant-chef
COOKBOOK_PATH=${PROJ_PATH}/cookbooks/hello_web
INSPEC_PATH=${COOKBOOK_PATH}/test/integration/default/cd ${COOKBOOK_PATH}mkdir -p ${INSPEC_PATH}
touch .kitchen.yml ${INSPEC_PATH}/default_test.rb

The updated structure will look like this with new additions in bold text:

~/vagrant-chef
└── cookbooks
    └── hello_web
        ├── .kitchen.yml
        ├── attributes
        │   └── default.rb
        ├── files
        │   └── default
        │       └── index.html
        ├── metadata.rb
        ├── recipes
        │   └── default.rb
        └── test
            └── integration
                └── default
                    └── default_test.rb

Create Test Kitchen Configuration

Update the .kitchen.yml with the following:

---
driver:
  name: vagrantprovisioner:
  name: chef_zero
  always_update_cookbooks: trueverifier:
  name: inspecplatforms:
  - name: ubuntu-16.04suites:
  - name: default
    run_list:
      - recipe[hello_web::default]

This is a basic Test Kitchen configuration file, which tells that globally for all systems that Vagrant will be used to create the systems, Chef Zero will be used to configure the systems, and that InSpec is used to verify the configuration. It also specifies one system called default to create for each platform, which for now is only an Ubuntu 16.04 Xenial Xerus platform.

Creating Inspec Tests

Let’s create our test script using InSpec. We have two parts, one that tests the configuration, which is not considered a good practice, but may be required for edge use cases and put here for illustrative purposes, and another that tests what is the purpose of our cookbook has promised to complete.

Update the default test test/integration/default/default_test.rb with the following:

######## TEST CONFIGURATION ########
describe package('apache2') do
  it { should be_installed }
enddescribe service('apache2') do
  it { should be_installed }
  it { should be_enabled }
  it { should be_running }
end######## TEST PROMISE ########
describe port(80) do
  it { should be_listening }
enddescribe http('localhost') do
  its('status') { should eq 200 }
  its('headers.Content-Type') { should include 'text/html' }
  its('body') { should include 'Hello World!' }
end

InSpec supports a variety of resources documented at:

https://www.inspec.io/docs/reference/resources/

For testing the configuration, we use package and service, which InSpec will understand how to test regardless of the system (exception for FreeBSD).

These tests generally are not consider best practices and have a low value, because they are really testing Chef itself and its ability to configure the system. As will illustrate later, it becomes difficult to support these types of tests.

The most important is testing the what our cookbook is promised to do, which is have a web service listening on port 80, and serving our “Hello World!” content.

Running The Tests

Now that Test Kitchen is configured and we have our tests, let’s try this out in separate steps.

kitchen create    # create the system
kitchen converge  # download chef and provision the system

Note: if there was a bad download for the chef installer, which is stashed in the ~/.kitchen/cache/ directory, Test Kitchen will fail with a stack trace that obfuscates the actual problem. You will have to purge the downloaded package, such as chef_14.3.37–1_amd64.deb, in order to get Test Kitchen working again.

After doing convergence, you may see something similar to this amongst the output:

Recipe: hello_web::default
  * apt_package[apache2] action install
    - install version 2.4.18-2ubuntu3.9 of package apache2
  * service[apache2] action enable (up to date)
  * service[apache2] action start (up to date)

With the system successfully up and provisioned, verify the results:

kitchen verify    # run inspec to test system

After running verify, you should see something that looks similar to this:

Version: (not specified)
Target:  ssh://vagrant@127.0.0.1:2202System Package apache2
   ✔  should be installed
Service apache2
   ✔  should be installed
   ✔  should be enabled
   ✔  should be running
Port 80
   ✔  should be listening
http GET on localhost
   ✔  status should eq 200
   ✔  headers.Content-Type should include "text/html"
   ✔  body should include "Hello World!"Test Summary: 8 successful, 0 failures, 0 skipped
       Finished verifying <default-ubuntu-1604> (0m0.43s).

Part II: Running More Platforms

Let’s add some new platforms to try out. Update the .kitchen.yml with the following:

---
driver:
  name: vagrantprovisioner:
  name: chef_zero
  always_update_cookbooks: trueverifier:
  name: inspecplatforms:
  - name: ubuntu-16.04
  - name: centos-7
  - name: freebsd-11.2suites:
  - name: default
    run_list:
      - recipe[hello_web::default]

Create and Configure New Systems

Take this out for a spin with:

kitchen create    # create the system
kitchen converge  # download chef and provision system

We can see because we have dynamic attributes, the systems are configured appropriately.

Verify New Systems

Also now that the systems work, do a verification:

kitchen verify    # run inspec to test system

This should show some failures, something similar to

System Package apache2
  ×  should be installed
  expected that `System Package apache2` is installed
Service apache2
  ×  should be installed
  expected that `Service apache2` is installed
  ×  should be enabled
  expected that `Service apache2` is enabled
  ×  should be running
  expected that `Service apache2` is running

The cookbook knows how install, start, and enable appropriate for all the platforms, but our tests do not know the actual package name or service name. We need to build in a way to tell our tests the package and service name to use.

Update Inspec Tests

One way we can have more flexibility in the package and service name is by supporting InSpec attributes (not to be confused with Chef attributes, totally unrelated).

Update the default_test.rb to look like the following:

######## ATTRIBUTES ########
PACKAGE = attribute('package', 
                    default: 'apache2',
                    description: 'package name'
                   )
                   
SERVICE = attribute('service', 
                    default: 'apache2', 
                    description: 'service name'
                   )######## TEST CONFIGURATION ########
describe package(PACKAGE) do
  it { should be_installed }
enddescribe service(SERVICE) do
  it { should be_installed }
  it { should be_enabled }
  it { should be_running }
end######## TEST PROMISE ########
describe port(80) do
  it { should be_listening }
enddescribe http('localhost') do
  its('status') { should eq 200 }
  its('headers.Content-Type') { should include 'text/html' }
  its('body') { should include 'Hello World!' }
end

We need to update the Test Kitchen configuration to pass in the appropriate attributes as well.

---
driver:
  name: vagrantprovisioner:
  name: chef_zero
  always_update_cookbooks: trueverifier:
  name: inspecplatforms:
  - name: ubuntu-16.04
  - name: centos-7
    verifier:
      attributes:
        package: httpd
        service: httpd
  - name: freebsd-11.2
    verifier:
      attributes:
        package: apache24
        service: apache24suites:
  - name: default
    run_list:
      - recipe[hello_web::default]

Run the Tests

kitchen verify

This will give us better results. With CentOS, we should get this:

System Package httpd
  ✔  should be installed
Service httpd
  ✔  should be installed
  ✔  should be enabled
  ✔  should be running

And with FreeBSD, we’ll see this:

System Package apache24
  ↺  The `package` resource is not supported on your OS yet.
Service apache24
  ✔  should be installed
  ✔  should be enabled
  ✔  should be running

Part III: Better Practices

We can see two things we’ve learned so far. First, we know how we can pass attributes from Test Kitchen to our InSpec tests, which is a useful facility. Secondly, we can see how testing specific configuration is expensive than just testing the promise. Testing the configuration should only be used for niche use cases, and generally just avoided.

Update Tests

Update the test (default_test.rb) to include only the promise.

######## TEST PROMISE ########
describe port(80) do
  it { should be_listening }
enddescribe http('localhost') do
  its('status') { should eq 200 }
  its('headers.Content-Type') { should include 'text/html' }
  its('body') { should include 'Hello World!' }
end

Update Kitchen Configuration

Update the Test Kitchen configuration (.kitchen.yml) to remove passing InSpec attributes as we no longer need this:

driver:
  name: vagrantprovisioner:
  name: chef_zero
  always_update_cookbooks: trueverifier:
  name: inspecplatforms:
  - name: ubuntu-16.04
  - name: centos-7
  - name: freebsd-11.2suites:
  - name: default
    run_list:
      - recipe[hello_web::default]

Run the tests

kitchen verify

The tests will all succeed.

Wrap Up

That’s all there is to it for testing easily testing cookbooks using integration testing, and how to use InSpec attributes should you need it.

Both tools are worth exploring further. Test Kitchen can use more than just Vagrant for managing the systems. It has plugins to support VMWare, GCE, EC2, Docker, Kubernetes, and others.

With InSpec, you can test system resources remotely, with not only SSH, but also WinRM and Docker, and of course locally as well. In addition to system resources, InSpec can test cloud resources themselves with GCP, AWS, Azure, and VMWare.

Testing Chef Cookbooks with InSpec

Integration Testing using Test Kitchen and Inspec

Prerequisites

Previous Guides

VirtualBox and Friends on Windows 8.1

VirtualBox, Vagrant, Test Kitchen, Docker Machine, Minikube

VirtualBox and Friends on macOS

VirtualBox, Vagrant, Test Kitchen, Docker Machine, Minikube

Virtualbox and Friends on Fedora 28

VirtualBox, Vagrant, Test Kitchen, Docker Machine, Minikube

Install Test Kitchen and InSpec with Ruby (Advanced)

A Tale of Two Rubies, Part I

Managing Rubies with RVM

A Tale of Two Rubies, Part II

Managing Rubies with Rbenv

Three Rubies and a ChefDK

ChefDK requires that you use their embedded ruby for running the Chef tools, but you may already have an existing ruby…

Part I: Getting Started

Creating Chef Cookbook

Creating Test Testing Structure

Create Test Kitchen Configuration

Creating Inspec Tests

Running The Tests

Part II: Running More Platforms

Create and Configure New Systems

Verify New Systems

Update Inspec Tests

Run the Tests

Part III: Better Practices

Update Tests

Update Kitchen Configuration

Run the tests

Wrap Up

Written by Joaquín Menchaca (智裕)