Open in app

Sign in

Write

Sign in

HashiCorp Vault with AppRole

Securing Secrets using Vault and AppRole Auth

Joaquín Menchaca (智裕)
11 min readApr 19, 2021

--

Security is an essential and a core part of operations and thus keeping secrets secured is vital. Unfortunately, for many an organization, this is often not a priority.

The lack of zeal toward managing secrets is likely related toward the complexity involved. Managing configuration artifacts have well established patterns using change configuration (, , , ) tools, using service discovery with KV stores (, , ), or through simpler means like environment vars and config files.

When the configuration artifacts are secrets, called secrets artifacts, you have to not only encrypt the secrets, but also secure who or what has access to these secrets. If the secret is ever compromised, you need a centralized location to recycle the secret artifacts. For this problem, Hashicorp is one of the most popular solution for this.

Securing secrets with Vault

The secret artifacts can be secured in HashiCorp using the method.

In this guide, I will walk you through the following:

  • Setting up a server using †
  • Setting up store and the method
  • Creating a admin role that will be used to create secrets and an dgraph application role (see below)
  • Testing access to the secrets using the dgraph application role
  • Configuration the application (see below) to use the dgraph application role.
  • Walk-through in testing features (ACL and encrypted export and backup) enabled through vault configuration.

† NOTE: is used as an example to get the ball rolling. With this, you can see how the services fit together, and then with this knowledge move this to your own environment, such as , , or .

For this guide, the current stable version 1.7.0 will be used:

export VAULT_VERSION="1.7.0"

About Dgraph

For the demonstration application, this guide will use , a highly performant distributed graph database. Two features used are and . These features require secret artifacts to be saved on disk, which is well, not all that secure. Fortunately, supports fetching secret artifacts directly from .

has two types of servers: Dgraph Zero nodes ( 0) that control state of the cluster and Dgraph Alpha nodes ( 1) that are the actual database nodes. For simplicity, we will just use a single Dgraph Zero node and a single Dgraph Alpha node.

For this guide, the current stable version v21.03.0 will be used:

export DGRAPH_VERSION="v21.03.0"

Prerequisites

These instructions will use shell commands that are tested in .

On macOS, all these tools, including , can be easily installed with brew command (see ).

On Windows 10, the command line tools can be installed with pacman in the environment. Both and can be installed with the choco command (see ).

Docker Compose setup

Use the following docker-compose.yaml configuration for this guide.

docker-compose.yaml

For this configuration, save default environment variables for VAULT_VERSION and DGRAPH_VERSION can be saved in an .env file:

cat <<-EOF > .env
DGRAPH_VERSION=v21.03.0
VAULT_VERSION=1.7.0
EOF

Create two directories vault and dgraph to store configuration files. We can do this and populate with blank files that will be use later using :

mkdir -p {vault,dgraph}
touch vault/config.hcl \
      dgraph/{vault_secret_id,vault_role_id,alpha.yaml}

The file structure should now look like the following:

.
├── .env
├── dgraph
│   ├── alpha.yaml
│   ├── vault_role_id
│   └── vault_secret_id
├── docker-compose.yaml
└── vault
    └── config.hcl

Vault service

For the service, use the following configuration and save it as ./vault/config.hcl:

vault/config.hcl

Once ready, start the service, and only the vault service with:

## launch vault server
docker-compose up --detach "vault"

Unseal Vault Service

Once the service is ready, initialize a fresh new environment with:

## initialize vault and copy secrets down
docker exec -t vault vault operator init

Copy down the secrets from this process, and use it for the unsealing process.

## unseal vault using copied secrets
docker exec -ti vault vault operator unseal
docker exec -ti vault vault operator unseal
docker exec -ti vault vault operator unseal

When finished, in the local session, set these two environment variables up, using the root token from above. These two environment variables will be used in further steps.

export VAULT_ROOT_TOKEN="<root-token>"
export VAULT_ADDRESS="127.0.0.1:8200"

Configure Vault with root privileges

This guide will use ’s API rather than the vault command. This may seem more complex, but it is the easiest way to access the service and also demonstrates how to interact with the server.

These steps will use the root token (VAULT_ROOT_TOKEN) copied from earlier from initializing the server.

Create AppRole and KV Secrets

Configure :

curl --silent \
  --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
  --request POST \
  --data '{"type": "approle"}' \
  $VAULT_ADDRESS/v1/sys/auth/approle

Configure :

curl --silent \
  --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
  --request POST \
  --data '{ "type": "kv-v2" }' \
  $VAULT_ADDRESS/v1/sys/mounts/secret

Create the admin policy

We need to create a policy that establishes an admin persona. Save the file below as ./vault/policy_admin.hcl.

vault/policy_admin.hcl

We will need to package the policy file in format as . This can be done with sed (tested with ):

cat <<EOF > ./vault/policy_admin.json
{
  "policy": "$(sed -e ':a;N;$!ba;s/\n/\\n/g' \
                   -e 's/"/\\"/g' \
                   vault/policy_admin.hcl)"
}
EOF

Upload the admin policy:

curl --silent \
  --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
  --request PUT --data @./vault/policy_admin.json \
  http://$VAULT_ADDRESS/v1/sys/policies/acl/admin

Verify that this was a success:

curl --silent \
  --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
  --request GET \
  http://$VAULT_ADDRESS/v1/sys/policies/acl/admin | jq

Create the admin role

Now that the admin policy is uploaded, create the admin role that uses this policy for the admin persona using the root token:

## create the admin role with an attached policy
curl --silent \
  --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
  --request POST \
  --data '{ 
"token_policies": "admin", 
"token_ttl": "1h", 
"token_max_ttl": "4h" 
}' \
  http://$VAULT_ADDRESS/v1/auth/approle/role/admin

Verify the admin role was created successfully:

## verify the role
curl --silent \
  --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
  --request GET \
  http://$VAULT_ADDRESS/v1/auth/approle/role/admin | jq

Retreive Admin Token

For this point forward, we will use the admin persona to setup secrets and the application persona called dgraph. We need to retreive the admin token by logging into with the admin persona.

First fetch the admin role-id:

VAULT_ADMIN_ROLE_ID=$(curl --silent \
  --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
  http://$VAULT_ADDRESS/v1/auth/approle/role/admin/role-id \
    | jq -r '.data.role_id'
)

Then fetch the corresponding admin secret-id:

VAULT_ADMIN_SECRET_ID=$(curl --silent \
  --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
  --request POST \
  http://$VAULT_ADDRESS/v1/auth/approle/role/admin/secret-id \
    | jq -r '.data.secret_id'
)

Using the role id and secret id, get and admin token and save this as the environment variable VAULT_ADMIN_TOKEN:

export VAULT_ADMIN_TOKEN=$(curl --silent \
  --request POST \
  --data "{ \"role_id\": \"$VAULT_ADMIN_ROLE_ID\", \"secret_id\": \"$VAULT_ADMIN_SECRET_ID\" }" \
  http://$VAULT_ADDRESS/v1/auth/approle/login \
    | jq -r '.auth.client_token'
)

Configure Vault with admin persona

For this section, the admin token (VAULT_ADMIN_TOKEN) will be used to upload secrets and create the dgraph application policy and role.

Save Secrets

There are two secrets used, the encryption key and secret file, which can uploaded using example below (saved as ./vault/payload_alpha_secrets.json):

vault/payload_alpha_secrets

For the , we add a cas (create-and-set) operation to guard against overwriting existing secrets (see ). Any update to the secrets will require incrementing the cas number.

When ready, upload the secrets payload:

curl --silent \
  --header "X-Vault-Token: $VAULT_ADMIN_TOKEN" \
  --request POST \
  --data @./vault/payload_alpha_secrets.json \
  http://$VAULT_ADDRESS/v1/secret/data/dgraph/alpha | jq

Create Dgraph application policy

The application policy is limited to reading the secrets (save this as ./vault/policy_dgraph.hcl):

vault/policy_dgraph.hcl

The first step is to package the policy as JSON using sed (tested with ):

## convert policies to json format
cat <<EOF > ./vault/policy_dgraph.json
{
  "policy": "$(sed -e ':a;N;$!ba;s/\n/\\n/g' \ 
                   -e 's/"/\\"/g' \
                   vault/policy_dgraph.hcl)"
}
EOF

Once converted, upload the dgraph policy:

## create the dgraph policy
curl --silent \
  --header "X-Vault-Token: $VAULT_ADMIN_TOKEN" \
  --request PUT --data @./vault/policy_dgraph.json \
  http://$VAULT_ADDRESS/v1/sys/policies/acl/dgraph

Verify the dgraph policy was uploaded:

## verify the policy
curl --silent \
  --header "X-Vault-Token: $VAULT_ADMIN_TOKEN" \
  --request GET \
  http://$VAULT_ADDRESS/v1/sys/policies/acl/dgraph | jq

Create Dgraph application role

Create the dgraph application role:

## create the dgraph role with an attached policy
curl --silent \
 --header "X-Vault-Token: $VAULT_ADMIN_TOKEN" \
 --request POST \
 --data '{ "token_policies": "dgraph", "token_ttl": "1h", "token_max_ttl": "4h" }' \
 http://$VAULT_ADDRESS/v1/auth/approle/role/dgraph

Verify that the role was created:

## verify the role
curl --silent \
  --header "X-Vault-Token: $VAULT_ADMIN_TOKEN" \
  --request GET \
  http://$VAULT_ADDRESS/v1/auth/approle/role/dgraph | jq

Retreive the Dgraph application token

Fetch the dgraph role-id:

VAULT_DGRAPH_ROLE_ID=$(curl --silent \
  --header "X-Vault-Token: $VAULT_ADMIN_TOKEN" \
  http://$VAULT_ADDRESS/v1/auth/approle/role/dgraph/role-id \
    | jq -r '.data.role_id'
)

Fetch the dgraph secret-id:

VAULT_DGRAPH_SECRET_ID=$(curl --silent \
  --header "X-Vault-Token: $VAULT_ADMIN_TOKEN" \
  --request POST \
  http://$VAULT_ADDRESS/v1/auth/approle/role/dgraph/secret-id \
    | jq -r '.data.secret_id'
)

Retrieve and save the dgraph token as VAULT_DGRAPH_TOKEN:

export VAULT_DGRAPH_TOKEN=$(curl --silent \
  --request POST \
  --data "{ \"role_id\": \"$VAULT_DGRAPH_ROLE_ID\", \"secret_id\": \"$VAULT_DGRAPH_SECRET_ID\" }" \
  http://$VAULT_ADDRESS/v1/auth/approle/login \
    | jq -r '.auth.client_token'
)

Save the dgraph role-id and secret-id as files to be used with configuring :

echo $VAULT_DGRAPH_ROLE_ID > ./dgraph/vault_role_id
echo $VAULT_DGRAPH_SECRET_ID > ./dgraph/vault_secret_id

Verify Secrets with Dgraph application persona

Using the dgraph token (VAULT_DGRAPH_TOKEN), verify that secrets can be retreived:

curl --silent \
  --header "X-Vault-Token: $VAULT_DGRAPH_TOKEN" \
  --request GET \
  http://$VAULT_ADDRESS/v1/secret/data/dgraph/alpha | jq

Dgraph Service

The Dgraph Alpha service will need to be configured to use to fetch secrets for the encryption and ACL features. Save the following to ./dgraph/alpha.yaml:

dgraph/alpha.yaml

This configuration is relative to the Dgraph Alpha container, where file paths are mounted into the container, and DNS vault address is accessed from within the ’s implicit private network that is automatically created, which you can view with:

docker network inspect "${PWD##*/}_default" | jq '.[].Containers'

Bring the remaining services up with:

docker-compose up --detach

You can inspect the logs with:

docker logs alpha1

You will want to scan for ACL secret key loaded successfully, and Encryption feature enabled in the logs.

Check the health as well as list of features available by querying the /health path on Dgraph Alpha with:

# print a list of features enabled
curl --silent http://localhost:8080/health \
  | jq -r '.[].ee_features | .[]' \
  | sed 's/^/* /' \
  | grep --color --extended-regexp 'acl|encrypt.*|$'

The full feature list with features enabled by vault configuration in ./dgraph/alpha.yaml highlighted:

* acl
* multi_tenancy
* encryption_at_rest
* encrypted_backup_restore
* encrypted_export
* cdc

Testing Dgraph features

This area demonstrates how to directly test the Dgraph () and features when enabled through the vault configuration.

For the Dgraph () feature, we just need to simply login and retreive an access JWT token, which is required to trigger the other features. For the export and backup operations, we need to supply a the token with the X-Dgraph-AccessToken field in the header. If is not enabled, this token will be ignored and not used.

For Dgraph feature, we can verify encryption is working performing a backup or export operations of the database. From the file system, we should see that the contents are encrypted.

Testing Dgraph ACL Feature

The feature is only enabled if a secret was saved. We can tested it by using the default administrative user (groot) and password (password) with request to /login path:

DGRAPH_ADMIN_USER="groot"
DGRAPH_ADMIN_PSWD="password"
export DGRAPH_ALPHA_ADDRESS="localhost:8080"## login using RESTful API
export DGRAPH_ACCESS_TOKEN=$(curl --silent \
  --request POST \
  --data "{
    \"userid\": \"$DGRAPH_ADMIN_USER\", 
    \"password\": \"$DGRAPH_ADMIN_PSWD\", 
    \"namespace\": 0 
}" \ 
  http://$DGRAPH_ALPHA_ADDRESS/login \
    | grep -oP '(?<=accessJWT":")[^"]*'
)

When are enabled with , administrative tasks like or will need have an access JWT tokens.

The environment variables DGRAPH_ACCESS_TOKEN and DGRAPH_ALPHA_ADDRESS will be used for other tests.

Testing Dgraph export operation

The feature will export the whole database in or formats. When is enabled, the exported database will be encypted.

For this administrative chore, we will create a query and save it as ./dgraph/export.graphql:

dgraph/export.graphql

After this, using the access token from earlier, we can trigger an export to the /admin path.

curl --silent \
  --header "Content-Type: application/graphql" \
  --header "X-Dgraph-AccessToken: $DGRAPH_ACCESS_TOKEN" \
  --request POST \
  --upload-file ./dgraph/export.graphql \
   | jq

This will create an export file structure similar to:

./dgraph/export
└── dgraph.r8.u0419.1810
    ├── g01.gql_schema.gz
    ├── g01.json.gz
    └── g01.schema.gz

You can get information about these files with:

find ./dgraph/export/ -name '*.gz' \
  | xargs -n 1 file | awk -F/ '{print $NF}'

If these files are encrypted SUCCESSFULLY then the result should say:

g01.schema.gz: data
g01.json.gz: data
g01.gql_schema.gz: data

Otherwise (FAILURE), it will say something like the following:

g01.schema.gz: gzip compressed data, max speed, original size modulo 2^32 381
g01.json.gz: gzip compressed data, max speed, original size modulo 2^32 5
g01.gql_schema.gz: gzip compressed data, max speed, original size modulo 2^32 5

Testing Dgraph backup operation

The feature will, as expected, backup the database with either full or incremental backups. This can be used to restore the database to a cluster. When is enabled, the backups will be encrypted.

For this administrative chore, we will create a query and save it as ./dgraph/backup.graphql:

dgraph/backup.graphql

You can trigger a backup with the following command to /admin path:

curl --silent \
  --header "Content-Type: application/graphql" \
  --header "X-Dgraph-AccessToken: $DGRAPH_ACCESS_TOKEN" \
  --request POST \
  --upload-file ./dgraph/backup.graphql \
   | jq

When completed, this will create an file structure similar to:

./dgraph/backups
├── dgraph.20210419.181116.903
│   └── r8-g1.backup
└── manifest.json

You can get information about the backup files with:

find ./dgraph/backups/ -name '*.backup' \
 | xargs -n 1 file | awk -F/ '{print $NF}'

A result that is SUCCESSFUL will show something like the following:

r8-g1.backup: data

If this is not encrypted (FAILURE), then you would see something like this:

r8-g1.backup: snappy framed data

Announcement: Updated Article

I recently created updated version of this article.

One major thing has changed in the Docker platform is that the python tool (previously introduced as ) is deprecated and rewritten in go as a plugin for Docker.

Vault AppRole Auth: The Hard Way

This article mirrors this article in that the REST API is used.

Vault AppRole Auth: The Hard Way

Securely Storing Secrets using HashiCorp Vault REST API

joachim8675309.medium.com

Vault AppRole Auth: The Hard Way

This article uses the Vault CLI, but is otherwise similar to the content in this article.

Vault AppRole Auth: The Easy Way

Securely Storing Secrets with HashiCorp Vault

joachim8675309.medium.com

Resources

The source code files for this tutorial can be found from here:

Conclusion

There you have it, a full overiew of saving secrets using auth method and store using with an application service, , that has direct support for this feature.

If the application(s) you support do not have direct support for Hashicorp , you can use the methods presented here in your automation to get the same functionality, and then provide the secrets to the application in a format it understands, such as environment variables, configuration files, etc.

With the role, you can also use an (also called a whitelist in the industry), where you can filter in an address or range of addresses that can access the server. This can be an alternative method to the required secret-id or and additional form of screening along with the secret-id. For example (attention to bind_secret_id and bound_cidr_list keys):

curl --silent \
 --header "X-Vault-Token: $VAULT_ADMIN_TOKEN" \
 --request POST \
 --data '{
    "token_policies": "dgraph",
    "token_ttl": "1h",
    "token_max_ttl": "4h",
    "bind_secret_id": false,
    "bound_cidr_list": [
      "10.0.0.0/8",
      "172.0.0.0/8",
      "192.168.0.0/16",
      "127.0.0.1/32"
    ]
}' \

Beyond the auth method, there are several other auth methods, such as these:

In a follow-up article, I would like to explore deploying an HA Vault cluster using with popular cloud providers, such as , , and . With the cloud providers, you can use the cloud provider’s KMS to auto-unseal the secrets, without the need to run vault operator unseal several times on all of the vault nodes. If you use a shared backend, such as , you only need to initialize one node with vault operator init. In a self-managed scenario, you can also use small vault backend to unseal the main HA Vault cluster. This article will be useful to help springboard toward building a HA vault.

I hope this is useful to get your started on your Hashicorp journey. Best of wishes and success.

Vault
DevOps
Dgraph
Docker
Sre

--

--

Joaquín Menchaca (智裕)
Joaquín Menchaca (智裕)

Written by Joaquín Menchaca (智裕)

1.6K Followers
139 Following

DevOps/SRE/PlatformEng — k8s, o11y, vault, terraform, ansible

Responses (1)

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams