TestKitchen with Ansible and TestInfra

Testing Ansible provisioned systems with Test Kitchen

This article covers configuring systems with Ansible and testing systems using TestInfra, keeping everything within the Python umbrella. Test Kitchen is a powerful tool to connect all of these together to bring up systems, provision them, and verify they are configured to meet the promise of the Ansible role.

Overview of Ansible

Ansible is a robust remote execution system that sets out to be more intuitive and easy to use than the shell. Ansible does change configuration through a push based method rather than through a pull method which requires an agent to be installed on the system before it can be configured. This affords Ansible greater flexibility, which allows Ansible to facilitate deployment and orchestration of systems, coordinating configuration based on state of other systems.

Beyond configuring systems resources (packages, services, files), Ansible can configure cloud resources (gcp, aws, azure, etc) or any web resources. Systems and system resources can be configured based on the overall cloud infrastructure state.

This gives Ansible unparalleled integration capability, with the ability to achieve Infrastructure as Code that goes beyond single system change configuration of the past, and allows cloud infrastructure provisioning and orchestration.

Ansible scripts called playbooks use DSL (domain-specific language) in a YAML format. A system or infrastructure is configured using procedural steps called tasks. These tasks along with variables, files, and templates can all be packaged together into a component called a role.

This guide will use a role called hello_web to install an Apache server with “Hello World!” content.

Overview of Test Kitchen

Test Kitchen is a test harness that creates, destroys, provisions, and verifies systems. The systems are created using a backend driver, most common are Vagrant and Docker. When systems become available, Test Kitchen can then provision them through a converge process, using a popular change configuration solution like Ansible or Chef. After provisioning, you can verify the correctness of those changes through a verifier plugin.

The default verifier is the busser system, which will install the test tool on the local system, and verify it using local execution on the guest system. You can use other verifier plug-ins that do not use the busser system.

Previous Ansible Vagrant Guide

In a previous guide, I demonstrated how to use Vagrant to bring up a system and provision it with Ansible.

Vagrant Provisioning with Ansible

Prerequisites

Before we begin this journey, we need to install 3 to 7 components on your development workstation:

1. Bash (optional): instructions are written using Bash. You can convert the instructions to what makes sense for your environment.
2. Virtualbox: free open source virtual management solution for Windows, Linux, and macOS.
3. Vagrant: underlying tool that manages virtual guest systems.
4. Test Kitchen 1.23.0 and Kitchen-Ansible plugin: test harness that manages, provisions, and verifies test systems. This can be installed using ChefDK 3.2.30 or with Ruby.
5. TestInfra and Paramiko (optional): this is needed if you wish to use this test tool on the host (for Part 5).
6. Python 2.7 (optional) if you use shell provisioner with TestInfra, you will need Python.
7. Ruby 2.5.1 (optional) if you wish to install test-kitchen gem instead of using ChefDK, you will need to install Ruby.

Installing Test Kitchen with ChefDK

Test Kitchen can be easily installed with ChefDK, which contains all the tools needed for running Test Kitchen:

• https://downloads.chef.io/chefdk

Once ChefDK is install, run the following to get Kitchen-Ansible:

chef gem install kitchen-ansible.

Installing Test Kitchen with Ruby

If you are uncomfortable with installing ChefDK, and you have Ruby 2.5.1, you can install the components manually. You may want to use ruby version managers like RVM and rbenv. With ruby 2.5.1 installed, you can install required components with:

gem install test-kitchen
gem install kitchen-vagrant
gem install kitchen-ansible

Installing TestInfra

Python has their own ecosystem for managing Python language versions and modules. I recommend using a tool like pyenv to get a flexible and consistent experience for managing Python versions:

• https://github.com/pyenv/pyenv

Once you have pip tool installed, you can install the tools:

pip install testinfra
pip install paramiko

For further documentation, see:

• https://github.com/philpep/testinfra
• https://testinfra.readthedocs.io/en/latest/

Previous Installation Guides

I wrote up some articles for installing Virtualbox, Vagrant, and ChefDK on Windows, Linux, and macOS.

Windows 8.1 using Chocolatey:

VirtualBox and Friends on Windows 8.1

Linux (Fedora 28):

Virtualbox and Friends on Fedora 28

macOS High Sierra 10.3 using Homebrew:

VirtualBox and Friends on macOS

Previous Ruby Management Guides

Installing the desired Ruby version can be complex and there are a few Ruby managers out there, like rvm and rbenv. I wrote some previous guides for installing ruby managers that assist in installing ruby, the requirement needed for installing test kitchen and any other ruby tools and libraries.

RVM:

A Tale of Two Rubies, Part I

rbenv:

A Tale of Two Rubies, Part II

And a follow on article should you want to use one of these Ruby managers and ChefDK together:

Three Rubies and a ChefDK

Part I: Initial Setup

In this part we set up the basic structure and create the systems. We will use an Ansible role called hello_web to install Apache and add “Hello World!” content. This will then later be tested to make sure this works.

Tip: If you would like to fetch the Vagrant boxes used by Test Kitchen and required for this project beforehand, you can do this with the following:

vagrant box add bento/ubuntu-16.04
vagrant box add bento/centos-7
vagrant box add bento/freebsd-11.2

Creating the Structure

You can run this in bash to create the basic structure:

PROVISION_PATH=~/vagrant-ansible/provisioning
ROLE_PATH=${PROVISION_PATH}/roles/hello_web
mkdir -p ${ROLE_PATH}/{defaults,files,tasks}
touch ${ROLE_PATH}/{defaults,tasks}/main.yml  \
  ${PROVISION_PATH}/playbook.yml \
  ${ROLE_PATH}/files/index.html \
  ${ROLE_PATH}/.kitchen.yml
cd ${ROLE_PATH}

With this, we should have the following:

~/vagrant-ansible
└── provisioning
    ├── playbook.yml
    └── roles
        └── hello_web
            ├── .kitchen.yml
            ├── defaults
            │   └── main.yml
            ├── files
            │   └── index.html
            └── tasks
                └── main.yml

Initial Configuration

We need to setup an initial Test Kitchen configuration (.kitchen.yml) in the root of our hello_web role. Update the .kitchen.yml with the following:

We will also need to update the playbook.yml found in ~/vagrant-ansible/provisioning/.

---
- hosts: all
  gather_facts: yes
  become: true

Create Systems

Bring up the system, run:

kitchen create

If the vagrant box images were already downloaded, it will take roughly 45 seconds to stand up a single system.

Once the system is up, run a command on it to verify it is working:

kitchen exec ubuntu -c 'lsb_release -a'

Part 2: Converging with Ansible

Now let’s create add the configuration code and try it out.

Update Playbook

Update the playbook.yml for our future role:

---
- hosts: all
  gather_facts: yes
  become: true
  roles:
    - hello_web

Update Role Default Variables

Update the role default variables (hello_web/defaults/main.yml) that are suitable for Ubuntu:

---
hello_web:
  docroot: /var/www/html
  package: apache2
  service: apache2

Update File Content

Update the file content (hello_web/files/index.html) for our “Hello World!”:

<html>
  <body>
    <h1>Hello World!</h1>
  </body>
</html>

Update Role Tasks

Update the role tasks (hello_web/tasks/main.yml):

---
- name: "Install Web Service"
  package:
    name: "{{ hello_web.package }}"
    state: present
- name: "Start Web Service"
  service:
    name: "{{ hello_web.service }}"
    state: started
    enabled: yes
- name: "Copy Content"
  copy:
    src: "{{ role_path }}/files/index.html"
    dest: "{{ hello_web.docroot }}/index.html"

Provision Ubuntu System

Now that we have the hello_web role completed and a playbook that references the role, let’s run it:

kitchen converge

This process will take 2 minutes, where 20 seconds is used to configure the system, and the rest of the time is installing Ansible and Busser on the system guest systems.

Part 3: Verifying with TestInfra Busser

For testing, we will explore TestInfra for verification. There is direct integration with tool through the busser-testinfra gem, so we’ll start with this.

Create Test Infrastructure

PROVISION_PATH=~/vagrant-ansible/provisioning
ROLE_PATH=${PROVISION_PATH}/roles/hello_web
TEST_PATH=${ROLE_PATH}/test/integration/default/testinfra
mkdir -p ${TEST_PATH}
touch ${TEST_PATH}/test_default.py
cd ${ROLE_PATH}

This will give us the following structure:

~/vagrant-ansible
└── provisioning
    ├── playbook.yml
    └── roles
        └── hello_web
            ├── .kitchen.yml
            ├── defaults
            │   └── main.yml
            ├── files
            │   └── index.html
            ├── tasks
            │   └── main.yml
            └── test
                └── integration
                    └── default
                        └── testinfra
                            └── test_default.py

Update Kitchen Configuration

Update the Test Kitchen configuration (.kitchen.yml) with the following below. with a lifecycle hook to install pip, which is required for busser-testinfra to work.

There are two additions provisioner option and lifecycle hook is needed for this process to work.

The provisioner option of require_chef_for_busser tells the ansible_playbook provisioner to install chef, which bundles the busser command needed for any busser verifier. Once this is in place, busser will then automatically in install the busser-testinfra gem.

The lifecycle hooks are a recent feature, which allows us to install pip before we attempt a verify. The pip tool for installing Python modules is required, because the busser-testinfra assumes it is installed. With this requirement made, busser-testinfra will install testinfra tool before running tests.

Create Test Script

For this test, it is very simple, test the promise of the role to create a web service listening at port 80.

Update the test_default.py test script with the following:

def test_port_80_is_listening(host):
  socket = host.socket("tcp://80")
  assert(socket.is_listening)

Verify the Ubuntu System

Now that we have our test script, let’s verify our system.

kitchen verify

When running this the first time, there will be 20 seconds needed to install the busser-testinfra gem and the TestInfra tool on the guest systems. Once the prerequisites are installed, it takes about 3 seconds to run the tests locally on each system.

Part 4: Adding Multiple Platforms

Testing only a single system of Ubuntu is not adventurous enough, so to spice things up, let’s add more systems to configuration.

Adding CentOS

First new system to play with is CentOS, update the following below needed to support CentOS with Ansible:

First, the package and service name are different on CentOS and Ubuntu, so we need to override the defaults by passing them to the ansible-playbook command, which are passed in using extra_vars key.

For TestInfra installed through busser-testinfra, we need need to install pip and the six module as well.

For some reason, with CentOS, there are scp sync issues, and for whatever the reason, adding this max_ssh_sessions key will workaround the issue.

Verify CentOS

With everything set, we can go ahead create the guest system and provision CentOS:

kitchen create centos
kitchen converge centos

If these all look good, we can then verify the system:

kitchen verify centos

Now we have two guests configured with Ansible and tested with TestInfra.

Adding FreeBSD

To add support for FreeBSD, we need to update Test Kitchen config (.kitchen.yml) to the following:

For FreeBSD, the kitchen-ansible plug-in does not support FreeBSD, so we are on our own for installing Ansible, and the requirements for TestInfra. We can do this before we do a converge with the lifecycle hook.

When running Ansible, in addition to overriding the default package, service, and docroot variables (as they are different for FreeBSD), we also need to tell ansible where to find python if it is not in the default /usr/bin/python path. We can do this by supplying the variable ansible_python_interpreter, which Ansible will use.

Once we completed the necessary prerequisites, we can bring up and configure the systems with Ansible:

kitchen create freebsd
kitchen converge freebsd

And then install TestInfra and test the system:

kitchen verify freebsd

Now with we have configured and tested all three systems.

Part 5: TestInfra from Host with Shell Provisioner

Thus far we have configured three systems and added the necessary glue to make it work. As part of this, we also installed Busser and TestInfra on the systems and ran tests locally on each system.

As an alternative to all of this, we could use TestInfra installed on the host instead of the guest, and use it’s ability to remote into the systems and turn the tests.

Configure Shell Verifier to use TestInfra

To facilitate this, we can use the shell verifier and call TestInfra on the host with Test Kitchen supplied environment variables that allow us to remote into the guest system and run tests:

In this configuration, as we not using busser, we can turn it off. Also, as we no longer need to install TestInfra on the guests, we an remove the lifecycle hooks needed to support local testing.

Using the shell verifier, we can now easily test systems using TestInfra with the py.test command.

Verify using Shell Verifier

With all of this, you can now run the test and see the results:

kitchen verify

Running the tests from the hosts takes about 2 to 3 seconds to run on each system, roughly the same about of time to run locally.

Conclusion

This tutorial was a bit of a challenging exercise, not because of Ansible itself, but rather Test Kitchen plugins (kitchen-ansible or busser-testinfra) were unable to install required components. Fortunately, with the new lifecycle hooks feature, such problems were ameliorated.

Running Ansible in local mode is fine for smoke testing a few roles, but if you need to test automation that uses orchestration and coordination amongst a set of systems, local mode system provisioning is inadequate.

For more advance multi-node scenarios, you may have to explore other options, where Ansible is running on the host using dynamic inventory script, like vagrant.py script, in combination with a test harness like Molecule. For everything else, Test Kitchen is fine.