Open in app

Sign in

Write

Sign in

HashiCorp Vault with AppRole

Securing Secrets using Vault and AppRole Auth

Joaquín Menchaca (智裕)
11 min readApr 19, 2021

--

Security is an essential and a core part of operations and thus keeping secrets secured is vital. Unfortunately, for many an organization, this is often not a priority.

The lack of zeal toward managing secrets is likely related toward the complexity involved. Managing configuration artifacts have well established patterns using change configuration (Puppet, Chef, Ansible, Salt Stack) tools, using service discovery with KV stores (etcd, Consul, Zookeeper), or through simpler means like environment vars and config files.

When the configuration artifacts are secrets, called secrets artifacts, you have to not only encrypt the secrets, but also secure who or what has access to these secrets. If the secret is ever compromised, you need a centralized location to recycle the secret artifacts. For this problem, Hashicorp Vault is one of the most popular solution for this.

Securing secrets with Vault

The secret artifacts can be secured in HashiCorp Vault using the AppRole method.

In this guide, I will walk you through the following:

  • Setting up a Vault server using Docker Compose†
  • Setting up KV Secrets v2 store and the AppRole method
  • Creating a admin role that will be used to create secrets and an dgraph application role (see below)
  • Testing access to the secrets using the dgraph application role
  • Configuration the application Dgraph (see below) to use the dgraph application role.
  • Walk-through in testing features (ACL and encrypted export and backup) enabled through Dgraph vault configuration.

† NOTE: Docker Compose is used as an example to get the ball rolling. With this, you can see how the services fit together, and then with this knowledge move this to your own environment, such as Systemd, Swarm, or Kubernetes.

For this guide, the current stable version 1.7.0 will be used:

export VAULT_VERSION="1.7.0"

About Dgraph

For the demonstration application, this guide will use Dgraph, a highly performant distributed graph database. Two features used are Encryption At Rest and Access Control Lists. These features require secret artifacts to be saved on disk, which is well, not all that secure. Fortunately, Dgraph supports fetching secret artifacts directly from Vault.

Dgraph has two types of servers: Dgraph Zero nodes (Raft consensus group 0) that control state of the cluster and Dgraph Alpha nodes (Raft consensus group 1) that are the actual database nodes. For simplicity, we will just use a single Dgraph Zero node and a single Dgraph Alpha node.

For this guide, the current stable version v21.03.0 will be used:

export DGRAPH_VERSION="v21.03.0"

Prerequisites

These instructions will use shell commands that are tested in Bash.

On macOS, all these tools, including Docker Destkop, can be easily installed with brew command (see Homebrew).

On Windows 10, the command line tools can be installed with pacman in the MSYS2 environment. Both MSYS2 and Docker Desktop can be installed with the choco command (see Chocolatey).

Docker Compose setup

Use the following docker-compose.yaml configuration for this guide.

docker-compose.yaml

For this configuration, save default environment variables for VAULT_VERSION and DGRAPH_VERSION can be saved in an .env file:

cat <<-EOF > .env
DGRAPH_VERSION=v21.03.0
VAULT_VERSION=1.7.0
EOF

Create two directories vault and dgraph to store configuration files. We can do this and populate with blank files that will be use later using Bash:

mkdir -p {vault,dgraph}
touch vault/config.hcl \
      dgraph/{vault_secret_id,vault_role_id,alpha.yaml}

The file structure should now look like the following:

.
├── .env
├── dgraph
│   ├── alpha.yaml
│   ├── vault_role_id
│   └── vault_secret_id
├── docker-compose.yaml
└── vault
    └── config.hcl

Vault service

For the Vault service, use the following configuration and save it as ./vault/config.hcl:

vault/config.hcl

Once ready, start the Vault service, and only the vault service with:

## launch vault server
docker-compose up --detach "vault"

Unseal Vault Service

Once the Vault service is ready, initialize a fresh new Vault environment with:

## initialize vault and copy secrets down
docker exec -t vault vault operator init

Copy down the secrets from this process, and use it for the unsealing process.

## unseal vault using copied secrets
docker exec -ti vault vault operator unseal
docker exec -ti vault vault operator unseal
docker exec -ti vault vault operator unseal

When finished, in the local session, set these two environment variables up, using the root token from above. These two environment variables will be used in further steps.

export VAULT_ROOT_TOKEN="<root-token>"
export VAULT_ADDRESS="127.0.0.1:8200"

Configure Vault with root privileges

This guide will use Vault’s RESTful API rather than the vault command. This may seem more complex, but it is the easiest way to access the Vault service and also demonstrates how to interact with the Vault server.

These steps will use the root token (VAULT_ROOT_TOKEN) copied from earlier from initializing the Vault server.

Create AppRole and KV Secrets

Configure AppRole:

curl --silent \
  --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
  --request POST \
  --data '{"type": "approle"}' \
  $VAULT_ADDRESS/v1/sys/auth/approle

Configure KV Secrets v2:

curl --silent \
  --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
  --request POST \
  --data '{ "type": "kv-v2" }' \
  $VAULT_ADDRESS/v1/sys/mounts/secret

Create the admin policy

We need to create a policy that establishes an admin persona. Save the file below as ./vault/policy_admin.hcl.

vault/policy_admin.hcl

We will need to package the policy file in HCL format as JSON. This can be done with sed (tested with GNU sed):

cat <<EOF > ./vault/policy_admin.json
{
  "policy": "$(sed -e ':a;N;$!ba;s/\n/\\n/g' \
                   -e 's/"/\\"/g' \
                   vault/policy_admin.hcl)"
}
EOF

Upload the admin policy:

curl --silent \
  --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
  --request PUT --data @./vault/policy_admin.json \
  http://$VAULT_ADDRESS/v1/sys/policies/acl/admin

Verify that this was a success:

curl --silent \
  --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
  --request GET \
  http://$VAULT_ADDRESS/v1/sys/policies/acl/admin | jq

Create the admin role

Now that the admin policy is uploaded, create the admin role that uses this policy for the admin persona using the root token:

## create the admin role with an attached policy
curl --silent \
  --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
  --request POST \
  --data '{ 
"token_policies": "admin", 
"token_ttl": "1h", 
"token_max_ttl": "4h" 
}' \
  http://$VAULT_ADDRESS/v1/auth/approle/role/admin

Verify the admin role was created successfully:

## verify the role
curl --silent \
  --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
  --request GET \
  http://$VAULT_ADDRESS/v1/auth/approle/role/admin | jq

Retreive Admin Token

For this point forward, we will use the admin persona to setup secrets and the application persona called dgraph. We need to retreive the admin token by logging into Vault with the admin persona.

First fetch the admin role-id:

VAULT_ADMIN_ROLE_ID=$(curl --silent \
  --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
  http://$VAULT_ADDRESS/v1/auth/approle/role/admin/role-id \
    | jq -r '.data.role_id'
)

Then fetch the corresponding admin secret-id:

VAULT_ADMIN_SECRET_ID=$(curl --silent \
  --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
  --request POST \
  http://$VAULT_ADDRESS/v1/auth/approle/role/admin/secret-id \
    | jq -r '.data.secret_id'
)

Using the role id and secret id, get and admin token and save this as the environment variable VAULT_ADMIN_TOKEN:

export VAULT_ADMIN_TOKEN=$(curl --silent \
  --request POST \
  --data "{ \"role_id\": \"$VAULT_ADMIN_ROLE_ID\", \"secret_id\": \"$VAULT_ADMIN_SECRET_ID\" }" \
  http://$VAULT_ADDRESS/v1/auth/approle/login \
    | jq -r '.auth.client_token'
)

Configure Vault with admin persona

For this section, the admin token (VAULT_ADMIN_TOKEN) will be used to upload secrets and create the dgraph application policy and role.

Save Secrets

There are two secrets used, the encryption key and HMAC secret file, which can uploaded using example JSON below (saved as ./vault/payload_alpha_secrets.json):

vault/payload_alpha_secrets

For the KV Secrets v2, we add a cas (create-and-set) operation to guard against overwriting existing secrets (see Writing/Reading arbitrary data). Any update to the secrets will require incrementing the cas number.

When ready, upload the secrets payload:

curl --silent \
  --header "X-Vault-Token: $VAULT_ADMIN_TOKEN" \
  --request POST \
  --data @./vault/payload_alpha_secrets.json \
  http://$VAULT_ADDRESS/v1/secret/data/dgraph/alpha | jq

Create Dgraph application policy

The application policy is limited to reading the secrets (save this as ./vault/policy_dgraph.hcl):

vault/policy_dgraph.hcl

The first step is to package the policy as JSON using sed (tested with GNU sed):

## convert policies to json format
cat <<EOF > ./vault/policy_dgraph.json
{
  "policy": "$(sed -e ':a;N;$!ba;s/\n/\\n/g' \ 
                   -e 's/"/\\"/g' \
                   vault/policy_dgraph.hcl)"
}
EOF

Once converted, upload the dgraph policy:

## create the dgraph policy
curl --silent \
  --header "X-Vault-Token: $VAULT_ADMIN_TOKEN" \
  --request PUT --data @./vault/policy_dgraph.json \
  http://$VAULT_ADDRESS/v1/sys/policies/acl/dgraph

Verify the dgraph policy was uploaded:

## verify the policy
curl --silent \
  --header "X-Vault-Token: $VAULT_ADMIN_TOKEN" \
  --request GET \
  http://$VAULT_ADDRESS/v1/sys/policies/acl/dgraph | jq

Create Dgraph application role

Create the dgraph application role:

## create the dgraph role with an attached policy
curl --silent \
 --header "X-Vault-Token: $VAULT_ADMIN_TOKEN" \
 --request POST \
 --data '{ "token_policies": "dgraph", "token_ttl": "1h", "token_max_ttl": "4h" }' \
 http://$VAULT_ADDRESS/v1/auth/approle/role/dgraph

Verify that the role was created:

## verify the role
curl --silent \
  --header "X-Vault-Token: $VAULT_ADMIN_TOKEN" \
  --request GET \
  http://$VAULT_ADDRESS/v1/auth/approle/role/dgraph | jq

Retreive the Dgraph application token

Fetch the dgraph role-id:

VAULT_DGRAPH_ROLE_ID=$(curl --silent \
  --header "X-Vault-Token: $VAULT_ADMIN_TOKEN" \
  http://$VAULT_ADDRESS/v1/auth/approle/role/dgraph/role-id \
    | jq -r '.data.role_id'
)

Fetch the dgraph secret-id:

VAULT_DGRAPH_SECRET_ID=$(curl --silent \
  --header "X-Vault-Token: $VAULT_ADMIN_TOKEN" \
  --request POST \
  http://$VAULT_ADDRESS/v1/auth/approle/role/dgraph/secret-id \
    | jq -r '.data.secret_id'
)

Retrieve and save the dgraph token as VAULT_DGRAPH_TOKEN:

export VAULT_DGRAPH_TOKEN=$(curl --silent \
  --request POST \
  --data "{ \"role_id\": \"$VAULT_DGRAPH_ROLE_ID\", \"secret_id\": \"$VAULT_DGRAPH_SECRET_ID\" }" \
  http://$VAULT_ADDRESS/v1/auth/approle/login \
    | jq -r '.auth.client_token'
)

Save the dgraph role-id and secret-id as files to be used with configuring Dgraph:

echo $VAULT_DGRAPH_ROLE_ID > ./dgraph/vault_role_id
echo $VAULT_DGRAPH_SECRET_ID > ./dgraph/vault_secret_id

Verify Secrets with Dgraph application persona

Using the dgraph token (VAULT_DGRAPH_TOKEN), verify that secrets can be retreived:

curl --silent \
  --header "X-Vault-Token: $VAULT_DGRAPH_TOKEN" \
  --request GET \
  http://$VAULT_ADDRESS/v1/secret/data/dgraph/alpha | jq

Dgraph Service

The Dgraph Alpha service will need to be configured to use Vault to fetch secrets for the encryption and ACL features. Save the following to ./dgraph/alpha.yaml:

dgraph/alpha.yaml

This configuration is relative to the Dgraph Alpha container, where file paths are mounted into the container, and DNS vault address is accessed from within the Docker Compose’s implicit private network that is automatically created, which you can view with:

docker network inspect "${PWD##*/}_default" | jq '.[].Containers'

Bring the remaining services up with:

docker-compose up --detach

You can inspect the logs with:

docker logs alpha1

You will want to scan for ACL secret key loaded successfully, and Encryption feature enabled in the logs.

Check the health as well as list of features available by querying the /health path on Dgraph Alpha with:

# print a list of features enabled
curl --silent http://localhost:8080/health \
  | jq -r '.[].ee_features | .[]' \
  | sed 's/^/* /' \
  | grep --color --extended-regexp 'acl|encrypt.*|$'

The full feature list with features enabled by vault configuration in ./dgraph/alpha.yaml highlighted:

* acl
* multi_tenancy
* encryption_at_rest
* encrypted_backup_restore
* encrypted_export
* cdc

Testing Dgraph features

This area demonstrates how to directly test the Dgraph ACL (access control lists) and encyption at rest features when enabled through the vault configuration.

For the Dgraph ACL(access control lists) feature, we just need to simply login and retreive an access JWT token, which is required to trigger the other features. For the export and backup operations, we need to supply a the token with the X-Dgraph-AccessToken field in the header. If ACL is not enabled, this token will be ignored and not used.

For Dgraph encyption at rest feature, we can verify encryption is working performing a backup or export operations of the Dgraph database. From the file system, we should see that the contents are encrypted.

Testing Dgraph ACL Feature

The ACL feature is only enabled if a secret was saved. We can tested it by using the default administrative user (groot) and password (password) with RESTful request to /login path:

DGRAPH_ADMIN_USER="groot"
DGRAPH_ADMIN_PSWD="password"
export DGRAPH_ALPHA_ADDRESS="localhost:8080"## login using RESTful API
export DGRAPH_ACCESS_TOKEN=$(curl --silent \
  --request POST \
  --data "{
    \"userid\": \"$DGRAPH_ADMIN_USER\", 
    \"password\": \"$DGRAPH_ADMIN_PSWD\", 
    \"namespace\": 0 
}" \ 
  http://$DGRAPH_ALPHA_ADDRESS/login \
    | grep -oP '(?<=accessJWT":")[^"]*'
)

When ACLs are enabled with Dgraph, administrative tasks like binary backups or exports will need have an access JWT tokens.

The environment variables DGRAPH_ACCESS_TOKEN and DGRAPH_ALPHA_ADDRESS will be used for other tests.

Testing Dgraph export operation

The export feature will export the whole Dgraph database in RDF or JSON formats. When encyption at rest is enabled, the exported database will be encypted.

For this administrative chore, we will create a GraphQL query and save it as ./dgraph/export.graphql:

dgraph/export.graphql

After this, using the access token from earlier, we can trigger an export to the /admin path.

curl --silent \
  --header "Content-Type: application/graphql" \
  --header "X-Dgraph-AccessToken: $DGRAPH_ACCESS_TOKEN" \
  --request POST \
  --upload-file ./dgraph/export.graphql \
  http://$DGRAPH_ALPHA_ADDRESS/admin | jq

This will create an export file structure similar to:

./dgraph/export
└── dgraph.r8.u0419.1810
    ├── g01.gql_schema.gz
    ├── g01.json.gz
    └── g01.schema.gz

You can get information about these files with:

find ./dgraph/export/ -name '*.gz' \
  | xargs -n 1 file | awk -F/ '{print $NF}'

If these files are encrypted SUCCESSFULLY then the result should say:

g01.schema.gz: data
g01.json.gz: data
g01.gql_schema.gz: data

Otherwise (FAILURE), it will say something like the following:

g01.schema.gz: gzip compressed data, max speed, original size modulo 2^32 381
g01.json.gz: gzip compressed data, max speed, original size modulo 2^32 5
g01.gql_schema.gz: gzip compressed data, max speed, original size modulo 2^32 5

Testing Dgraph backup operation

The binary backups feature will, as expected, backup the Dgraph database with either full or incremental backups. This can be used to restore the database to a Dgraph cluster. When encyption at rest is enabled, the backups will be encrypted.

For this administrative chore, we will create a GraphQL query and save it as ./dgraph/backup.graphql:

dgraph/backup.graphql

You can trigger a backup with the following command to /admin path:

curl --silent \
  --header "Content-Type: application/graphql" \
  --header "X-Dgraph-AccessToken: $DGRAPH_ACCESS_TOKEN" \
  --request POST \
  --upload-file ./dgraph/backup.graphql \
  http://$DGRAPH_ALPHA_ADDRESS/admin | jq

When completed, this will create an file structure similar to:

./dgraph/backups
├── dgraph.20210419.181116.903
│   └── r8-g1.backup
└── manifest.json

You can get information about the backup files with:

find ./dgraph/backups/ -name '*.backup' \
 | xargs -n 1 file | awk -F/ '{print $NF}'

A result that is SUCCESSFUL will show something like the following:

r8-g1.backup: data

If this is not encrypted (FAILURE), then you would see something like this:

r8-g1.backup: snappy framed data

Announcement: Updated Article

I recently created updated version of this article.

One major thing has changed in the Docker platform is that the python tool docker-compose (previously introduced as fig) is deprecated and rewritten in go as a plugin for Docker.

Vault AppRole Auth: The Hard Way

This article mirrors this article in that the REST API is used.

Vault AppRole Auth: The Hard Way

Securely Storing Secrets using HashiCorp Vault REST API

joachim8675309.medium.com

Vault AppRole Auth: The Hard Way

This article uses the Vault CLI, but is otherwise similar to the content in this article.

Vault AppRole Auth: The Easy Way

Securely Storing Secrets with HashiCorp Vault

joachim8675309.medium.com

Resources

The source code files for this tutorial can be found from here:

Conclusion

There you have it, a full overiew of saving secrets using AppRole auth method and KV Secrets v2 store using with an application service, Dgraph, that has direct support for this feature.

If the application(s) you support do not have direct support for Hashicorp Vault, you can use the methods presented here in your automation to get the same functionality, and then provide the secrets to the application in a format it understands, such as environment variables, configuration files, etc.

With the AppRole role, you can also use an bound CIDR list (also called a whitelist in the industry), where you can filter in an address or range of addresses that can access the server. This can be an alternative method to the required secret-id or and additional form of screening along with the secret-id. For example (attention to bind_secret_id and bound_cidr_list keys):

curl --silent \
 --header "X-Vault-Token: $VAULT_ADMIN_TOKEN" \
 --request POST \
 --data '{
    "token_policies": "dgraph",
    "token_ttl": "1h",
    "token_max_ttl": "4h",
    "bind_secret_id": false,
    "bound_cidr_list": [
      "10.0.0.0/8",
      "172.0.0.0/8",
      "192.168.0.0/16",
      "127.0.0.1/32"
    ]
}' \
 http://$VAULT_ADDRESS/v1/auth/approle/role/dgraph

Beyond the AppRole auth method, there are several other auth methods, such as these:

In a follow-up article, I would like to explore deploying an HA Vault cluster using Kubernetes with popular cloud providers, such as EKS, GKE, and AKS. With the cloud providers, you can use the cloud provider’s KMS to auto-unseal the secrets, without the need to run vault operator unseal several times on all of the vault nodes. If you use a shared backend, such as Consul, you only need to initialize one node with vault operator init. In a self-managed Kubernetes scenario, you can also use small vault backend to unseal the main HA Vault cluster. This article will be useful to help springboard toward building a HA vault.

I hope this is useful to get your started on your Hashicorp Vault journey. Best of wishes and success.

Vault
DevOps
Dgraph
Docker
Sre

--

--

Joaquín Menchaca (智裕)
Joaquín Menchaca (智裕)

Written by Joaquín Menchaca (智裕)

1.6K Followers
138 Following

DevOps/SRE/PlatformEng — k8s, o11y, vault, terraform, ansible

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams