Open in app

Sign in

Write

Sign in

Vault AppRole Auth: The Hard Way

Securely Storing Secrets using HashiCorp Vault REST API

Joaquín Menchaca (智裕)
14 min readMay 1, 2024

--

In a previous article, I demonstrated how to configure Hashicorp Vault to securely store secrets using the Vault AppRole authentication method, which uses role identities that are suited for automation. I included how to integrate this setup with applications like Dgraph, a high-performance distributed graph database that supports Vault natively.

In this article, I will guide you through the same procedure using only the REST API. This approach not only helps in understanding how to build Vault-supported applications but also serves as a comparative analysis to the similar operations performed using the Vault CLI.

Previous Article

Vault AppRole Auth: The Easy Way

Securely Storing Secrets with HashiCorp Vault

joachim8675309.medium.com

Requirements

These are the tool requirements used for this tutorials

Tool Requirements

Tested Environments

This was tested on the following environments

macOS Monterey 12.6.3 build 21G419
--------------------------------------------------
* Docker Desktop for macOS 4.29.0
  * Docker Engine 26.0.0
    * Plugin: Compose v2.26.1
* zsh 5.9 (arm-apple-darwin21.3.0)
* GNU bash, version 5.2.21(1)-release (aarch64-apple-darwin21.6.0)
* grep (GNU grep) 3.11
* sed (GNU sed) 4.9
* jq 1.7.1
* Vault v1.16.2

Windows 11 Home [WinNT 10.0.22631.34467] with MSYS
--------------------------------------------------
* Docker Desktop for Windows 4.29.0
  * Docker Engine 26.0.0
    * Plugin: Compose: v2.26.1
* MSYS
  * zsh 5.9 (x86_64-pc-msys)
  * GNU bash, version 5.2.26(1)-release (x86_64-pc-msys)
  * grep (GNU grep) 3.0
  * sed (GNU sed) 4.9
  * jq 1.7.1
* Vault v1.16.1

Pop!_OS 22.04 LTS (Ubuntu Jammy)
--------------------------------------------------
* Docker Engine version 26.1.0, build 9714adc
  * Plugin: Compose version v2.26.1
* zsh 5.8.1 (x86_64-ubuntu-linux-gnu)
* GNU bash, version 5.1.16(1)-release (x86_64-pc-linux-gnu)
* grep (GNU grep) 3.7
* sed (GNU sed) 4.8                                 
* jq 1.6
* Vault v1.16.2

Windows Tips

You can install tools such as Docker Desktop, Hashicorp Vault, and MSYS2 using the Chocolatey package management system with these commands in PowerShell with the Run As Administrator option:

choco install msys2 
choco install vault 
choco install docker-desktop

After setting up MSYS2, launch the MSYS2 environment and run the following to update packages and install jq:

# update packages
pacman -Suy
# install jq
pacman -S mingw-w64-x86_64-jq

To make the commands vault, jq, and vault available, we need to setup a new path:

DOCKER_PATH="/c/Program Files/Docker/Docker/resources/bin/"
CHOCO_PATH="/c/ProgramData/chocolatey/bin"
MINGW_PATH="/mingw64/bin/"

export PATH=$PATH:$MINGW_PATH:$CHOCO_PATH:$DOCKER_PATH

macOS Tips

If you can Homebrew installed, you can install the Docker Desktop, Hashicorp Vault, GNU grep, GNU Sed, and jq.

# Docker Desktop
brew -cask install docker
# Hashicorp Vault
brew tap hashicorp/tap
brew install hashicorp/tap/vault
# GNU Grep and jq
brew install grep gnu-sed jq

GNU Grep and GNU Sed will have to be configured in the PATH to take precedent over the limited BSD grep:

PATH="$HOMEBREW_PREFIX/opt/grep/libexec/gnubin:$PATH"
export PATH="$HOMEBREW_PREFIX/opt/gnu-sed/libexec/gnubin:$PATH"

Ubuntu Tips

Debian, Ubuntu, and most other Linux distributions are equipped with a standard POSIX environment, which includes GNU Bash, GNU Sed, and GNU Grep. For tools like curl and jq, you can install these with the following on Debian based distros like Ubuntu:

sudo apt-get update && sudo apt install curl jq

For Vault and Docker with the Compose plugin, visit these sites for further instructions:

Project Setup

This setup will setup the configuration for this tutorial

Setup Docker Compose

Run the following below to setup the compose environment.

## Setup Compose Env configuration
cat << EOF > .env
DGRAPH_VERSION=v23.1.1
VAULT_VERSION=1.16
EOF

## Setup Compose configuration
cat << 'EOF' > compose.yml
services:
  zero1:
    image: dgraph/dgraph:${DGRAPH_VERSION}
    command: dgraph zero --my=zero1:5080 --replicas 1 --raft idx=1
    ports:
      - 6080:6080
    container_name: zero1
  alpha1:
    image: dgraph/dgraph:${DGRAPH_VERSION}
    ports:
      - 8080:8080
      - 9080:9080
    environment:
      DGRAPH_ALPHA_CONFIG: /dgraph/config/config.yaml
    volumes:
      - ./dgraph/alpha.yaml:/dgraph/config/config.yaml
      - ./dgraph/vault_secret_id:/dgraph/vault/secret_id
      - ./dgraph/vault_role_id:/dgraph/vault/role_id
      - ./dgraph/backups:/dgraph/backups
      - ./dgraph/export:/dgraph/export
    command: dgraph alpha --my=alpha1:7080 --zero=zero1:5080
    container_name: alpha1
  vault:
    image: hashicorp/vault:${VAULT_VERSION}
    container_name: vault
    ports:
      - 8200:8200
    volumes:
      - ./vault/config.hcl:/vault/config/config.hcl
      - ./vault/data:/vault/data
    environment:
      VAULT_ADDR: http://127.0.0.1:8200
    entrypoint: vault server -config=/vault/config/config.hcl
    cap_add:
      - IPC_LOCK
EOF

Setup Vault Server

The config.hcl will configure the Vault service.

mkdir -p ./vault/data

cat << EOF > ./vault/config.hcl
storage "raft" {
  path    = "/vault/data"
  node_id = "vault1"
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = "true"
}

api_addr      = "http://127.0.0.1:8200"
cluster_addr  = "http://127.0.0.1:8201"
ui            = true
disable_mlock = true
EOF

Setup the Dgraph Server

The alpha.yaml will configure the Dgraph database service to utilize Vault to store secrets that are necessary to enable ACLs and encryption on disk.

mkdir -p ./dgraph/{backups,export}

cat << EOF > ./dgraph/alpha.yaml
vault:
  addr: http://vault:8200
  acl_field: hmac_secret_file
  acl_format: raw
  enc_field: enc_key
  enc_format: raw
  path: secret/data/dgraph/alpha
  role_id_file: /dgraph/vault/role_id
  secret_id_file: /dgraph/vault/secret_id
security:
  whitelist: 10.0.0.0/8,172.0.0.0/8,192.168.0.0/16
EOF

Launch Vault

Using Docker Compose, we can start up the Vault server with the following commands.

📓 IMPORTANT: GNU grep needs to be installed for PCRE (Perl Compatible Regular Expressions) with the -oP switch.

## launch vault server
docker compose up --detach "vault"

## configure client access
export VAULT_ADDR="http://127.0.0.1:8200"

## unseal
vault operator init | tee unseal.creds
## assume only 3 attempts are needed
for NUM in {1..3}; do
  vault operator unseal \
    $(grep -oP "(?<=Unseal Key $NUM: ).*" unseal.creds) > /dev/null
done

## export the results for use in other steps
export VAULT_ROOT_TOKEN="$(
  grep -oP "(?<=Initial Root Token: ).*" unseal.creds
)"

Configure Vault AppRole

Below are instructions to setup necessary role identities and access in Vault so that Dgraph can retrieve secrets.

Enable Auth

## enabled approle at approle/
curl --silent --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
  --request POST --data '{"type": "approle"}' \
  $VAULT_ADDR/v1/sys/auth/approle

You can list the auth methods enabled and search for the key approle/:

curl --silent --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
  $VAULT_ADDR/v1/sys/auth | jq -r '.data'

This should show something similar to this on a fresh server:

Create Policies

The policies define access to resources within Vault that can be assigned to roles. Run the following below to create a Dgraph admin policy and a restricted dgraph policy.

## Dgraph Policy
cat << EOF > ./vault/policy_dgraph.hcl
path "secret/data/dgraph/*" {
  capabilities = [ "read", "update" ]
}
EOF

## Convert HCL to JSON
cat <<EOF > ./vault/policy_dgraph.json
{
  "policy": "$(sed -e ':a;N;$!ba;s/\n/\\n/g' \
                   -e 's/"/\\"/g' \
                   ./vault/policy_dgraph.hcl)"
}
EOF

## Admin Policy
cat <<-EOF > ./vault/policy_admin.hcl
# kv2 secret/dgraph/*
path "secret/data/dgraph/*" {
   capabilities = [ "create", "read", "update", "delete", "list" ]
}

path "secret/metadata/dgraph/*" {
   capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Mount the AppRole auth method
path "sys/auth/approle" {
  capabilities = [ "create", "read", "update", "delete", "sudo" ]
}

# Configure the AppRole auth method
path "sys/auth/approle/*" {
  capabilities = [ "create", "read", "update", "delete" ]
}

# Create and manage roles
path "auth/approle/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Write ACL policies
path "sys/policies/acl/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}
EOF

## Convert HCL to JSON
cat << EOF > ./vault/policy_admin.json
{
  "policy": "$(sed -e ':a;N;$!ba;s/\n/\\n/g' \
                   -e 's/"/\\"/g' \
                   ./vault/policy_admin.hcl)"
}
EOF

# Upload Policies
curl --silent --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
  --request PUT --data @./vault/policy_admin.json \
  $VAULT_ADDR/v1/sys/policies/acl/admin

curl --silent --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
  --request PUT --data @./vault/policy_dgraph.json \
  $VAULT_ADDR/v1/sys/policies/acl/dgraph

You can verify the policy with

curl --silent --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
  $VAULT_ADDR/v1/sys/policies/acl/admin | jq .data.policy \
  | sed -r -e 's/\\n/\n/g' -e 's/\\"/"/g' -e 's/^"(.*)"$/\1/'

This should show the following:

You can verify the dgraph policy with following command below

curl --silent --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
  $VAULT_ADDR/v1/sys/policies/acl/dgraph | jq .data.policy \
  | sed -r -e 's/\\n/\n/g' -e 's/\\"/"/g' -e 's/^"(.*)"$/\1/'

This should show something like the following:

Create Role Identities

Run the following to create the needed role identities:

##############
# create admin role
############################
curl --silent --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
  --request POST \
  --data \
'{
    "token_policies": "admin",
    "token_ttl": "1h",
    "token_max_ttl": "4h"
}' \
  $VAULT_ADDR/v1/auth/approle/role/admin

##############
# create dgraph role
############################
curl --silent --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
  --request POST \
  --data \
'{
    "token_policies": "dgraph",
    "token_ttl": "1h",
    "token_max_ttl": "4h"
}' \
  $VAULT_ADDR/v1/auth/approle/role/dgraph

You can verify the admin role was created

curl --silent --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
  $VAULT_ADDR/v1/auth/approle/role/admin | jq .data

This should show something like the following:

You can verify the dgraph role was created

curl --silent --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
  $VAULT_ADDR/v1/auth/approle/role/dgraph | jq .data

This should show something like:

Configure Vault KV

You can enable KV v2 by using the commands below. This activates a key-value secrets engine that supports secret versioning, which is crucial for accessing previous versions of a secret. This feature is important for restore operations, using an backup from earlier Dgraph clusters that may have been encrypted with an earlier secret.

## enable kv-v2 at secret/
curl --silent --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
  --request POST --data '{ "type": "kv-v2" }' \
  $VAULT_ADDR/v1/sys/mounts/secret

You can run the following using a jq query to remap a subset of the data:

curl --silent --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
  $VAULT_ADDR/v1/sys/mounts \
  | jq -r '.data 
     | to_entries 
     | map({key, value: {
        type: .value.type, 
        accessor: .value.accessor, 
        description: .value.description
       }})
   '

This should show something like the following:

Create Secrets

Use the admin role ID to test its privileges for creating secrets, in addition to actually creating the secrets:

##############
# login using admin role
############################
ROLE_ID=$(curl --silent --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
  $VAULT_ADDR/v1/auth/approle/role/admin/role-id \
  | jq -r '.data.role_id'
)

SECRET_ID=$(curl --silent --header "X-Vault-Token: $VAULT_ROOT_TOKEN" \
  --request POST \
  $VAULT_ADDR/v1/auth/approle/role/admin/secret-id \
  | jq -r '.data.secret_id'
)

export VAULT_ADMIN_TOKEN=$(curl --silent \
  --request POST \
  --data \
"{
    \"role_id\": \"$ROLE_ID\",
    \"secret_id\": \"$SECRET_ID\"
}" \
  $VAULT_ADDR/v1/auth/approle/login \
  | jq -r '.auth.client_token'
)

Now that you are logged in using the admin token, VAULT_ADMIN_TOKEN, you can test saving the secrets (preferably random secrets) to save into Vault.

##############
# write dgraph secrets using admin role
############################
randpasswd() {
  NUM=${1:-32}

  # macOS scenario
  if [[ $(uname -s) == "Darwin" ]]; then
    perl -pe 'binmode(STDIN, ":bytes"); tr/A-Za-z0-9//dc;' \
      < /dev/urandom | head -c $NUM
  else
    # tested with: GNU/Linux, Cygwin, MSys
    tr -dc 'a-zA-Z0-9' < /dev/urandom | fold -w $NUM | sed 1q
  fi
}

## create json with secrets
cat << EOF > ./payload_alpha_secrets.json
{
  "options": {
    "cas": 0
  },
  "data": {
    "enc_key": "$(randpasswd)",
    "hmac_secret_file": "$(randpasswd)"
  }
}
EOF

# create using a random secrets
curl --silent \
  --header "X-Vault-Token: $VAULT_ADMIN_TOKEN" \
  --request POST \
  --data @./payload_alpha_secrets.json \
  $VAULT_ADDR/v1/secret/data/dgraph/alpha | jq

# delete secrets in file
rm ./payload_alpha_secrets.json

Read Secrets using the Dgraph Role

Now that the secrets have been created, the retrieval needs to be tested using the dgraph role with more restrictive permissions.

##############
# login using dgraph role
############################
ROLE_ID=$(curl --silent --header "X-Vault-Token: $VAULT_ADMIN_TOKEN" \
  $VAULT_ADDR/v1/auth/approle/role/dgraph/role-id \
    | jq -r '.data.role_id'
)

SECRET_ID=$(curl --silent --header "X-Vault-Token: $VAULT_ADMIN_TOKEN" \
  --request POST \
  $VAULT_ADDR/v1/auth/approle/role/dgraph/secret-id \
    | jq -r '.data.secret_id'
)

export VAULT_DGRAPH_TOKEN=$(curl --silent \
  --request POST \
  --data "{ \"role_id\": \"$ROLE_ID\", \"secret_id\": \"$SECRET_ID\" }" \
  $VAULT_ADDR/v1/auth/approle/login \
    | jq -r '.auth.client_token'
)

Using the VAULT_DGRAPH_TOKEN, you should be able to query the stored secrets with the following command:

curl --silent --header "X-Vault-Token: $VAULT_DGRAPH_TOKEN" \
  $VAULT_ADDR/v1/secret/data/dgraph/alpha | jq

This should show something like that the following:

Now that the secrets are working, we can save them role id and secret id that we fetched earlier for the dgraph role.

# test access to secrets using dgraph role credentials
curl --silent --header "X-Vault-Token: $VAULT_DGRAPH_TOKEN" \
  $VAULT_ADDR/v1/secret/data/dgraph/alpha > /dev/null
if [[ $? == 0 ]]; then
  echo $ROLE_ID > ./dgraph/vault_role_id
  echo $SECRET_ID > ./dgraph/vault_secret_id
fi

Launch Dgraph

Now that the necessary secrets have been stored in Vault and we’ve verified access to the credentials using the dgraph role-id, we are ready to launch Dgraph.

Execute the following commands to start Dgraph Zero followed by Dgraph Alpha.

##############
# launch Dgraph Zero
############################
docker compose up --detach "zero1"

##############
# launch Dgraph Alpha if credentials exist
############################
if [[ -f "./dgraph/vault_role_id" && -f "./dgraph/vault_secret_id" ]]; then
  docker compose up --detach "alpha1"
fi

You can check the logs to make sure there are now errors:

docker logs alpha1

You can also look through the logs to see if the ACL secret key loaded successfully and that the ENcryption feature is enabled:

docker logs alpha1 2>&1 \
 | grep --color -E 'ACL secret key|Encryption feature'

This should show something like the following:

You can also see the features enabled by querying the health of Dgraph Alpha with the following command:

curl --silent http://localhost:8080/health \
  | jq -r '.[].ee_features | .[]' \
  | sed 's/^/* /' \
  | grep --color --extended-regexp 'acl|encrypt.*|$'

This should show something:

Login into Dgraph

Now that ACLs are activated, Dgraph will need an access token for interaction. You can log in using the root identity with the initial default password or password.

DGRAPH_ADMIN_USER="groot"
DGRAPH_ADMIN_PSWD="password"
export DGRAPH_HTTP="localhost:8080"

export DGRAPH_TOKEN=$(curl --silent \
  --request POST \
  --data "{
    \"userid\": \"$DGRAPH_ADMIN_USER\", 
    \"password\": \"$DGRAPH_ADMIN_PSWD\", 
    \"namespace\": 0 
}" \
  http://$DGRAPH_HTTP/login | grep -oP '(?<=accessJWT":")[^"]*'
)

📓 IMPORTANT: For both production and even testing environments, it’s vital to change the default password for the root identity ‘groot’. Vault can be used to securely store and manage this password. For proper access control, you will want to set up an administrative account by creating a new user and assigning them to the ‘guardians’ group. Additionally, create regular user accounts that lack administrative privileges to further secure system access.

Testing Dgraph

When interacting with Dgraph, you will need to add X-Dgraph-AccessToken to the header with the token that was previously acquired from a login operation. More information regarding Access Control Lists from Dgraph documentation.

Getting Started Tutorial

If you would like to try out some of the basic features, I have a Getting Started Tutorial from a article that I wrote earlier. Even though this article was explicitly written for Kubernetes, the Getting Started Tutorial will work on either Docker and Kubernetes, as the Dgraph is available on localhost.

Dgraph on Kubernetes

Getting Started with Dgraph on Kubernetes

joachim8675309.medium.com

When going through this tutorial, and don’t forget to add the access token to the header. For example, here’s one of the queries in that tutorial with the additional header:

curl "$DGRAPH_HTTP/query" --silent --request POST \
  --header "X-Dgraph-AccessToken: $DGRAPH_TOKEN" \
  --header "Content-Type: application/dql" \
  --data $'{ me(func: has(starring)) { name } }' \
  | jq .data

Testing an Export Operation

The export feature will export the whole Dgraph database in RDF or JSON formats. When encyption at rest is enabled, the exported database will be encypted.

For this administrative chore, we will create a GraphQL query.

# Construct export mutation query
cat << EOF > ./dgraph/export.graphql
mutation {
  export(input: { format: "json" }) {
    response {
      message
      code
    }
  }
}
EOF

# Issue Export Operation
curl --silent \
  --header "Content-Type: application/graphql" \
  --header "X-Dgraph-AccessToken: $DGRAPH_TOKEN" \
  --request POST \
  --upload-file ./dgraph/export.graphql \
  http://$DGRAPH_HTTP/admin | jq

This will create an export file structure similar to:

You can find further information regarding these files with:

find ./dgraph/export/ -name '*.gz' \
  | xargs -n 1 file \
  | awk -F/ '{print $NF}'

This should show something like the following if the files were encrypted:

Testing a Backup Operation (Enterprise feature)

The binary backups feature will, as expected, backup the Dgraph database with either full or incremental backups. This can be used to restore the database to a Dgraph cluster. When encyption at rest is enabled, the backups will be encrypted.

# Construct backup mutation query
cat << EOF > ./dgraph/backup.graphql
mutation {
  backup(input: {
    destination: "/dgraph/backups"
    forceFull: true
  }) {
    response {
      message
      code
    }
  }
}
EOF

# Issue Binary backup operation
curl --silent \
  --header "Content-Type: application/graphql" \
  --header "X-Dgraph-AccessToken: $DGRAPH_TOKEN" \
  --request POST \
  --upload-file ./dgraph/backup.graphql \
  http://$DGRAPH_HTTP/admin | jq

When completed, this will create an file structure similar to:

You can get information about the backup files with:

find ./dgraph/backups/ -name '*.backup' \
 | xargs -n 1 file \
 | awk -F/ '{print $NF}'

A result that is SUCCESSFUL will show something like the following:

Cleanup

# remove allocated Docker resources 
docker compose stop && docker compose rm
# remove persisted vault data
rm -rf ./vault/data/*

Resources

Source Code

Conclusion

This tutorial provides a comprehensive guide on setting up and managing HashiCorp Vault the hard way, through the more challenging approach of using the web REST API. It details how to configure a client application, Dgraph, to store secrets in Vault utilizing the AppRole authentication method.

This setup enables critical Dgraph features such as access control lists and encryption at rest. Through this, this tutorial offers a secure way to manage automated services and applications, making it highly relevant for modern, automated, cloud-based architectures.

Some key steps covered in this tutorial include:

  1. Environment Setup: The tutorial begins with detailed instructions on setting up Docker, Vault, and necessary command-line tools across macOS, Windows, and Linux platforms, ensuring users from different environments can follow along.
  2. Vault and Dgraph Configuration: You learn to configure Vault with appropriate policies and roles for secure authentication and secret management. The configuration of Dgraph to utilize Vault for storing secrets necessary for ACL and encryption is also detailed.
  3. Securing Secrets with AppRole: This section focuses on using Vault’s AppRole for secure application-to-vault authentication, emphasizing secure storage and handling of secrets which is crucial for automated processes in production environments.
  4. Operational Commands: The tutorial provides commands for initializing and unsealing Vault, configuring roles and policies, and securing secret paths. It explains the use of curl and jq to interact with Vault's API, highlighting the transition from using Vault CLI to API for automation compatibility.
  5. Practical Implementation: Through detailed steps to set up, authenticate, and utilize roles, the guide demonstrates practical use cases such as generating, storing, and accessing secrets. It also covers the setup of Dgraph to use these secrets, ensuring database security.
  6. Testing and Validation: Steps to test and validate the configuration ensure that the implementations are secure and functional. These include logging into Dgraph with secured credentials and executing backup and export operations to verify that security measures like encryption are effectively in place.

This tutorial not only enhances understanding of HashiCorp Vault and Dgraph integrations but also equips readers with the knowledge to implement secure, automated data management solutions in a variety of operational environments.

Devsecops
Hashicorp Vault
Dgraph
Docker
Docker Compose

--

--

Joaquín Menchaca (智裕)

Written by Joaquín Menchaca (智裕)

1.5K Followers

DevOps/SRE/PlatformEng — k8s, o11y, vault, terraform, ansible

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams