Vagrant Provisioning with Puppet

Provisioning Virtual System using Puppet Apply

Joaquín Menchaca (智裕)
9 min readAug 13, 2018

--

Vagrant has two provisioners for Puppet, one that requires a puppet server, called puppet_server, and one that doesn’t, called puppet. This tutorial will use the later. In this brief tutorial, we’ll walk through how to setup a minimal Vagrant and Puppet environment and creating a Puppet module for a Apache web server. This will show how to use the puppet provisioner, which uses puppet apply on the command line.

About Puppet

Puppet is a centralized change configuration solution introduced by Luke Kaines in 2005 around theme of puppeteering several systems. The Puppet architecture uses a centralized Puppet Master server to communicate with distributed puppet agents installed on client nodes.

The configuration scripts used in Puppet are called manifests, and these can be organized into a component called a module. Puppet manifest use an intuitive proprietary DSL language oriented configuring resources, where resource is a “fundamental unit for modeling system configurations” and where “each resource describes some aspect of a system, like a specific service or package” (reference on resources).

The traditional starting point in Puppet is the site manifest, where you create node definitions that match on hostnames and define a list of classes that you wish to run on the matching system. For this tutorial, we’ll use the default node, which matches all systems.

Prerequisites

Before getting started you need to have Vagrant and Virtualbox installed. These instructions use Curl and Bash shell, so if this is not your shell, you will need to convert the shell commands to something that makes sense in your environment.

I have published some previous guides on installing Vagrant and Virtualboxon macOS, Windows, and Linux operating systems.

Windows 8.1

Using Chocolatey to install the requirements:

macOS High Sierra 10.3

Using Homebrew to install the requirements:

Fedora 28

Using native package manager Dandified YUM, or in short DNF, to install requirements:

Part I: Puppet Module with Intelligent Defaults

In this part, we’re going to create a Puppet module called hello_web that will install an Apache web server and copy over HTML content on Ubuntu system.

Create Staging Area

You can use these commands (Bash shell with touch) to create the structure for this tutorial:

mkdir -p ~/vagrant-puppet/{site,manifests}
cd ~/vagrant-puppet
mkdir -p site/hello_web/{files,manifests}
touch Vagrantfile bootstrap.sh manifests/default.pp \
site/hello_web/manifests/init.pp
cat <<-'HTML' > site/hello_web/files/index.html
<html>
<body>
<h1>Hello World!</h1>
</body>
</html>
HTML

This will create the following structure:

~/vagrant-puppet
├── Vagrantfile
├── bootstrap.sh
├── manifests
│ └── default.pp
└── site
└── hello_web
├── files
│ └── index.html
└── manifests
└── init.pp

Vagrant configuration

Update the Vagrantfile with the following below to use the Ubuntu box from the Bento project:

Vagrant.configure("2") do |config|
config.vm.box = "bento/ubuntu-16.04"
config.vm.network "forwarded_port", guest: 80, host: 8084
####### Install Puppet Agent #######
config.vm.provision "shell", path: "./bootstrap.sh"
####### Provision #######
config.vm.provision "puppet" do |puppet|
puppet.module_path = "./site"
puppet.options = "--verbose --debug"
end
end

Vagrant does not currently bootstrap, or rather install puppet agent. Thus, in order to use the Puppet Provisioner, we need to use a Vagrant box that already has puppet agent pre-installed (not advisable as this gets into the golden image anti-pattern), or we have to install it. For this tutorial, we’re going to use the shell provisioner to install the Puppet agent.

For the Puppet Provisioner, we specify where to source our modules, which is from a local directory ./site/. We’ll use the default site manifest location, which is ./manifests/default.pp.

We also turn on verbosity and debug so we can peak at what puppet is doing.

Bootstrap Puppet Agent

As mentioned, we need to install Puppet Agent before using the Puppet Provisioner. I created this install script and tested it against recent versions of CentOS, Fedora, Debian, and Ubuntu.

Site Manifest

For our site manifest, we’ll use the default node definition, which means all systems, all systems being the one virtual system. Update the ./manifests/default.pp to match this:

node default {
class { 'hello_web': }
}

The term class here represents the module. Alternative to this syntax, if you are not passing parameters into the class, you could simply do this:

node default {
include hello_web
}

Module Class

Now it is time to create the main course: the module’s class definition. The hello_web class will accept three parameters that are defaulted to some intelligent defaults that make sense for an Ubuntu system.

In the module under site/hello_web, update the manifests/init.pp:

class hello_web (
$package_name = 'apache2',
$service_name = 'apache2',
$doc_root = '/var/www/html'
) {
package { $package_name:
ensure => present,
}
service { $service_name:
ensure => running,
enable => true,
}
file { "$doc_root/index.html":
source => "puppet:///modules/hello_web/index.html",
}
}

Some background on defaulting variables. Puppet doesn’t actually have variables, they are really constants, but Puppet calls them variables. If you want to have something that acts like a real variable, where it can be overriden, you use parameters in Puppet. In the parameter list, you can set them to a desired default. For this reason, many modules may have rather large parameter lists.

For the resources being configured, Puppet’s DSL follows this pattern:

type { 'title':
attribute => value
}

The service and package resources are pretty straight forward, but the the file resource in Puppet is something special. It does many operations, one of them is to copy files from the files directory in the Puppet module to the desired destination. To reference this, puppet uses the syntax:

"puppet://modules/<your_module_name>/<your_file_name>"

This will map to the following below, where $MODULE_PATH is ./site/ in our case.

$MODULE_PATH/<your_module_name>/files/<your_file_name>

Test the Solution

We can download the virtual box image (if not done before), create the virtual guest, then first install puppet using the shell provisioner, and finally provision our system using puppet apply command through the Puppet provisioner:

vagrant up
curl -i http://127.0.0.1:8084

This will give us something like this:

HTTP/1.1 200 OK
Date
: Sun, 12 Aug 2018 14:04:06 GMT
Server: Apache/2.4.18 (Ubuntu)
Last-Modified: Sun, 12 Aug 2018 14:02:40 GMT
ETag: "3c-5733d6e1783d6"
Accept-Ranges: bytes
Content-Length: 60
Content-Type: text/html
<html>
<body>
<h1>Hello World!</h1>
</body>
</html>

Part II: Pass Parameters to Puppet Module

Shifting gears, let’s swap out previous system for CentOS from the Bento project. The default values specified in the parameter list will not work well for CentOS, so we’ll need to do something about that later.

Update Vagrant Configuration

First, let’s update the Vagrantfile to use CentOS.

Vagrant.configure("2") do |config|
config.vm.box = "bento/centos-7.5"
config.vm.network "forwarded_port", guest: 80, host: 8084
####### Install Puppet Agent #######
config.vm.provision "shell", path: "./bootstrap.sh"
####### Provision #######
config.vm.provision "puppet" do |puppet|
puppet.module_path = "./site"
puppet.options = "--verbose --debug"
end
end

Update Site Manifest

One way we can get new values into the module that support CentOS that override the defaults is by passing new values them into the class. We need to update the node definition, and when referencing the class, we pass in new values, like this below.

Update the manifests/default.pp to match the following:

node default {
class { 'hello_web':
package_name => 'httpd',
service_name => 'httpd',
doc_root => '/var/www/html',
}
}

Note: For the this is tutorial, it is fine to manually rejigger the values in the node definition, but out there in the real world, this is an anti-pattern and something we should never do, as humans are required for the automation to constantly change values, and thus won’t be automation anymore.

Test the Update

Once the Vagrantfile is updated and the new site manifest, purge the old environment and create a new virtual guest, provision it, and test the results with the following:

vagrant destroy --force
vagrant up
curl -i http://127.0.0.1:8084

This should give us something like:

HTTP/1.1 200 OK
Date
: Sun, 12 Aug 2018 14:56:32 GMT
Server: Apache/2.4.6 (CentOS)
Last-Modified: Sun, 12 Aug 2018 14:56:14 GMT
ETag: "3c-5733e2d9ffceb"
Accept-Ranges: bytes
Content-Length: 60
Content-Type: text/html; charset=UTF-8
<html>
<body>
<h1>Hello World!</h1>
</body>
</html>

Alternative Strategy (Intermediate)

In this tutorial, we use the default node, which means that anything we put in there will be applied to all nodes. This doesn’t give us a lot of flexibility. And is something that though fine for the tutorial or homegrown development environments, is something we would never do.

As alternatives, there are at least two strategies, we can classify our nodes or have smarter parameters..

Node Classification with Node Definitions

For the first method, we would want to assign our virtual guest a hostname, such as dk-ord-web01, representing, using some imagination, a web server in a Chicago datacenter that uses CentOS. To do this, we would update the Vagrantfile with the following:

Vagrant.configure("2") do |config|
config.vm.box = "bento/centos-7.5"
config.vm.hostname = "dk-ord-web01"
config.vm.network "forwarded_port", guest: 80, host: 8082
####### Install Puppet Agent #######
config.vm.provision "shell", path: "./bootstrap.sh"
####### Provision #######
config.vm.provision "puppet" do |puppet|
puppet.module_path = "./site"
puppet.options = "--verbose --debug"
end
end

To use this, you at minimum would need to do vagrant reload to assign the hostname, but better would be to remove it with vagrant destroy, so we can start with a clean virtual guest with vagrant up --no-provision.

Now that we have a desired hostname assigned to the virtual guest, we can then reference this in our manifest below.

Here’s two node definitions, borrowing some creative imagination, one definition for CentOS web servers in Chicago data center, and another for Ubuntu web servers in Las Vegas data center:

node dk-ord-web* {
class { 'hello_web':
package_name => 'httpd',
service_name => 'httpd',
doc_root => '/var/www/html',
}
}
node dk-las-web* {
class { 'hello_web':
package_name => 'apache2',
service_name => 'apache2',
doc_root => '/var/www/html',
}
}

Smart Parameters with inherited class

For the second method, we can use parameterization like champ with the params.pp pattern. The parameters are dynamically configured based on the operating system family, and then later referenced from our main class file.

To get started with this, create a ./site/hello_web/manifests/params.pp file that can self configure based on, for example, the OS family:

class hello_web::params {
case $::osfamily {
'Debian': {
$package_name = "apache2"
$service_name = "apache2"
$doc_root = "/var/www/html"
}
'RedHat': {
$package_name = "httpd"
$service_name = "httpd"
$doc_root = "/var/www/html"
}
}
}

Once this is in place, we can refer to them in our modules class init.pp:

class hello_web(
$package_name = $hello_web::params::package_name,
$service_name = $hello_web::params::service_name,
$doc_root = $hello_web::params::doc_root
) inherits hello_web::params {
package { $package_name:
ensure => present,
}
service { $service_name:
ensure => running,
enable => true,
}
file { "$doc_root/index.html":
source => "puppet:///modules/hello_web/index.html",
}
}

This will inherit the params class so that we can access the values, and then copy them from whatever the params configures, and installed them as defaults to our main class.

To demonstrate this working, revert the site manifest ./manifests/default.pp to the basics:

node default {
include hello_web
}

Now using either CentOS (bento/centos-7.5) or Ubuntu (bento/ubuntu-16.04) for the VM box set in the Vagrantfile, bring up a new system:

vagrant destroy --force
vagrant up
curl -i 127.0.0.1:8084

Updated Article

This article is published in 2018, and since then some things have changed. This is an updated article:

Final Thoughts

There you have it, how to utilize Vagrant for Puppet development using the Puppet provisioner. We walked through the basic concepts of resources, node definitions, and using modules, and how to make scripts more adaptive if you need to support different operating system families.

This is just the basic surface level of what is possible with Puppet. Puppet can be extended to support alternative mechanism to classify nodes beyond a Node manifest. This is done through the ENC (External Node Classifier) mechanism, where you can create your own solution, or use some of the popular ones.

In the Puppet community, currently systems are configured with an external classifier called Hiera, that uses a hierarchical structure to classify what classes are configured on a node. Hiera supports multiple backends, such as YAML, JSON, and databases.

This rich environment using Hiera is now the preferred method of configuration. I did not want to jump into Hiera just yet, as the idea of this tutorial is not to blow up newcomers to smithereens for an introductory tutorial.ca.conf

--

--

Joaquín Menchaca (智裕)
Joaquín Menchaca (智裕)

Written by Joaquín Menchaca (智裕)

DevOps/SRE/PlatformEng — k8s, o11y, vault, terraform, ansible

No responses yet