Creating GCE Instances with Knife
There may come a time where you would like a tool to conveniently create some systems on Google Cloud, and then apply some change configuration to it. With Chef, you can do this using the Knife-Google plug-in.
Why Use This?
There may be a few reasons why you may want to do this. One reason is when in a development or staging environment, you may have a need to create some quick systems without a lot of fuss. This tool helps with that.
Another reason, is that Chef currently has no mechanism to automate systems in groups, such as
prod-web-*, other than using automation at the bootstrap level.
You can configure a set of systems with an initial base state (security policies, standard package list, monitoring agents, other agents) and optionally the full chef run with a configured runlist.
Why Not Use This?
Personally, I don’t recommend mixing configuration of the infrastructure layer with instance or system layer. If you do this it becomes challenging to apply principals Infrastructure As Code as it will be more difficult and riskier to maintain the infrastructure. It would be best to do these separately with just
gcloud command line (or another tool) for the infrastructure layer and vanilla
knife bootstrap for the instance or system layer.
You definitely don’t want changes made in ad-hoc fashion outside of a control system (such as code checked into a git repository). For development environments in the cloud, this may be alright, at least where allocated resources are routinely recycled, thus avoiding cost creep. For a production environment, it is better to have some safeguards and controls, all document all changes through versioned code.
There are some use cases where you absolutely cannot use knife-google, such as situations where you need a set of systems created first, before you apply change configuration, knife-google will not work for this. This is because knife-google is an all or nothing sort of deal with each system, once you type
knife google server create, it will create the instance, and then immediately apply the bootstrap process.
A good example of this is an ElasticSearch cluster, where each node in the cluster must be configured with a list of IP addresses for all the others nodes in the cluster. In this situation, you would need to create the systems first in and compile a list of their IP addresses, and then configure the systems using the completed list.
Getting Started with Knife-Google
Before getting started, you need to have the following done already:
- Google Account with Access to create GCE instances in a Google Project.
- GCloud SDK installed and with credentials installed
- Chef Server account with config/credentials (
knife.rb) downloaded into your chef-repository (or appropriate path)
- Deployment SSH key pair with public key installed into your Google Project, and private key available on your workstation.
- Latest ChefDK installed (which is ChefDK 2.5.3 as of April 18th, 2018)
- Knife-Google plugin installed:
chef gem install knife-google
The current version of knife-google plug-in 3.2.0 will only work with ChefDK 2.5.3, and will crash (stacktrace) with earlier versions of ChefDK.
For creating the SSH key pair, I documented this process in this article:
Creating Some Systems
This will create three ElasticSearch nodes, one node per availability zone, add the tag of
my_cluster and some metadata, explicitly telling the system to use SSH keys previously installed into the Google project’s metadata.
After creating each system, it will then bootstrap that system. The bootstrap process will do the following:
- Install chef agent on the remote system.
- Create a node object entry on the Chef Server.
- Authorize the remote system with credentials from the Chef Server.
- Do a chef run process using the configured run-list.