Testing Change Config
A bug that gets into the web application can be fixed and tested, and rolled out to production. But what if there’s a flaw in the infrastructure?
This cannot be simply plucked out and changed, as the everything is dependent on this. Suddenly, testing is becoming more and more important to avoid this worst case scenario.
But for Ops, there’s no fancy IDEs that can run test infrastructures locally, and most shops don’t have CI (Continuous Integration) test the integrated change code, as this requires creating infrastructure to validate the final solution.
Fortunately, there are a few tools out there.
The Tools: Unit Tests
The unit tests are a first line of defense in increasing coding maturity and early warning for defects. It is valuable for testing expected result from change configuration platform’s interface, which sometimes yield surprising results and have bugs themselves. Both Puppet and Chef have this for testing the convergence of resources:
- RSpec-Puppet: https://github.com/rodjek/rspec-puppet/
• RSpec Tutorial: http://rspec-puppet.com/tutorial/
- ChefSpec: https://docs.chef.io/chefspec.html
• Learn ChefSpec Tutorial: https://learn.chef.io/modules/unit-test-with-chefspec#/
The Tools: Integration Tests
Testing the usage resources is important, but what about putting it all together, testing if the underlying package did what it was suppose to do, or testing in conjunction with deploy code; or what about testing migrating change configuration platforms, or testing configuration that occurs at build and deploy time instead of during runtime, such as the case with Docker and Kubernetes.
This is where integration tests are important, as they test the final result. A simple integration, called smoke test, can test the functionality of a service. More robust integration tests can test client and server integration, and may include other tier pieces like a database if so desired.
There are a few tools that have been used for this. Many custom crafted solutions have been done with Vagrant for systems, and Docker (often with Docker-Compose, for containers. The tests using these systems are typically manual tests, but can use more formal test platforms like ServerSpec or InSpec.
Enter Test Kitchen
Test Kitchen is a robust test-harness and orchestration solution that works across many platforms. The solution brings several components together in a consistent way:
- drivers (providers) —Vagrant (default), Docker, Hyper-V, AWS EC2, GCP GCE , etc.
- platforms — linux distro, windows, etc.
- provisioners — Puppet, Chef, Ansible, and SaltStack
- transport — shell exec, ssh, winrm, or docker-exec with dokken
- verifier — ServerSpec, InSpec, BATS, or through shell exec TestInfra, or GOSS.
The test scripts that are executed across these systems as specified after change configuration was applied. You can use this for smoke tests (testing the service functionality) or together with client-server and other tiers if so desired. The level of integration is left to the imagination.
As infrastructure that is being used by customers is difficult to change or test, unit tests and integration tests are vital to verify functionality. There may be an initial upfront cost to development tests, called prevention costs, but downstream the costs savings are significant given the nature of infrastructure tech debt.
Chef and Puppet have some tools for unit testing of converged resources with their perspective DSL, and for integration testing you can use ServerSpec or InSpec in conjunction with a robust test-harness orchestration tool Test Kitchen. In addition to this tool, development and testing using Vagrant and Docker-Compose, can speed up development, and for Ansible users, Molecule can be used as an alternative for integration testing.
Test Well My Friends