Dynamic Terraform API tokens

Intro

There are times where solutions that consume Terraform require API access. This may be through a higher order system like a CI/CD tool or pipeline. It could also be something such as system that will pull statistics or metrics from an API. How can we generate role specific API keys? Vault’s new Terraform Secret Engine can help us here.

Before we get into that lets understand a couple of different token types

Token

User Tokens - each User can have a number of API tokens on their behalf. They will inherit all role based access based upon their identity.

Team Tokens - each team can an a single API token at a given time. This is designed for applies by a higher order system.

Organization Tokens - whilst only one at a time can be granted it is designed for automating team management, membership, and workspaces!

If you want to know more click here

Demo time

An example that will demonstrate how to setup the secrets engine in Vault then explore how something would retrieve a secret from Vault. We will do this via API and CLI.

The high level workflow is the same for either method and is performed in two parts

Administrative Workflow

Preparation
- Create Teams
- Create initial token
Authenticate to Vault
Create Terraform Secret Engine
Define a role for Pipelines
Define a role for Consumption
Define a user role for testing

Consumption Workflow

Authenticate to Vault
Retrieve Token
Test against API
Refresh lease
Validate revocation

Administration Workflow

What we are doing here is one time configuration for the setup of this particular pattern.

Preparation - Terraform

There are a few steps we need to perform to start this off. Using the same token or an individual user token we need to setup the connection for Vault. I would recommend making a team and then using a specific team token as these are lazy rotated. This would mean that the team that binds Vault to Terraform won’t break or expire but can be rotated as needed.

Creating Teams

Perform the following in Terraform

Create a team called vault-terraform-token
Create a gitlab-pipeline team
Ensure that both teams visibility is set to secret

To retrieve the team-idyou can select your newly created team and find the team-id in the URL or use the following on the CLI

curl \
  --header "Authorization: Bearer $TOKEN" \
  --header "Content-Type: application/vnd.api+json" \
  --request GET \
  https://app.terraform.io/api/v2/organizations/$ORG/teams \
| jq '.data[] | .id + " " + .attributes.name'

Note Please use the envvar $TOKEN with a token that can read all teams. Please define an $ORG. In this case mine is burkey.

The results should list all teams and jq will concatenate two trings. The first is the team id and then the team name.

"team-bUPL36Yf9yJ9cbsx burkey-read-only"
"team-L2oUtQkF2gH4rBt2 gitlab-pipeline"
"team-Aw9YmqQN8sow7JmU owners"
"team-un4EYK9U1h5k2gtV burkey-outputs-only"
"team-J7eAmiNTAWTysLb4 relevent-team-name"
"team-cPuDYa4B5VRrmDsD vault-terraform-token"

The value I am after here is team-L2oUtQkF2gH4rBt2. This will be used in our role creation.

Generate Terraform/Vault token

There is a need for an authorartive token that allows Vault to programmatically drive Terraform. This should not be a user token give they’re bound to a user and subject to a TTL. Generate this token by selecting the team vault-terraform-token and then under Team API Token. Create a token.

Store this token as an envvar with export TF_TOKEN=<TOKEN>.

Shopping list

You now should have the following created:

Group called gitlab-pipeline
Group called vault-terraform-token
A team token created for the group vault-terraform-token.

Now - onto the creation of the Secrets Engine.

Create the secrets engine

Here we will create the secrets engine vault secrets enable -path tfc terraform

I have decided to use the tfc path for this Secrets Engine due to the fact I have a self-hosted terraform which I want to separate with a different Secret Engine name.

Now that we have a team token we generated against vault-terraform-token we will use the envvar $TF_TOKEN we set earlier.

vault write tfc/config token=$TF_TOKEN

Now that the engine is created we can create a number of roles depending on token type.

Create the Roles

Now I am creating the role for the gitlab-pipeline team. I retrieved previously retrieved via the API team-L2oUtQkF2gH4rBt2 and stored it in $GHP_TOKEN. I am also giving it a TTL of 2 minutes.

vault write tfc/role/gitlab-pipeline team_id=$GHP_TOKEN ttl=2m
Success! Data written to: tfc/role/gitlab-pipeline

Ace! Let’s confirm this.

vault read tfc/role/gitlab-pipeline
Key        Value
---        -----
max_ttl    0s
name       gitlab-pipeline
team_id    team-L2oUtQkF2gH4rBt2
ttl        2m

CLI Secret Retrieval

The CLI method will be how humans and operators potentially interact with the terraform secrets engine.

vault read tfc/creds/burkey-org
Key             Value
---             -----
organization    burkey
role            burkey-org
team_id         n/a
token           WAjxRdizm084qA.atlasv1.vdSGXwGn9dl2Hyea2DJyVaub9EhMw3SUisD702QxJ3WsCpTdR4R0h1G6tnnwYBwTFok
token_id        at-Lv43GzevZK5mZycr

vault read tfc/creds/gitlab-pipeline
Key             Value
---             -----
organization    n/a
role            gitlab-pipeline
team_id         team-L2oUtQkF2gH4rBt2
token           DQ9Fx0W3EEV1Ow.atlasv1.aeAvKQu6elAeVm1allHbsryVYod9gpjSvZyZwX9QVGAm59yt13x0YnGRpq7lNLXwJSA
token_id        at-a8RCaYBVVtdDNaqJ

Now I need to rotate these credentials as they are valid! Cheeky people reading this may try to access my organization or team otherwise.

vault write -f tfc/rotate-role/burkey-org && vault write -f tfc/rotate-role/gitlab-pipeline

API Secret Retrieval

The API approach will be how other machines will retrieve their tokens for Terraform from Vault.

Here we retrieve the current token for the team role for gitlab-pipeline. I then pipe it to jq for easier reading and manipulation.

curl \
  --header "X-Vault-Token: $VAULT_TOKEN" \
  --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
  --request GET \
  $VAULT_ADDR/v1/tfc/creds/gitlab-pipeline | jq

  <SNIP>

{
  "request_id": "2f45ecf0-adeb-068d-0ad0-62a734f6d173",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": {
    "organization": "",
    "role": "gitlab-pipeline",
    "team_id": "team-L2oUtQkF2gH4rBt2",
    "token": "DQ9Fx0W3EEV1Ow.atlasv1.aeAvKQu6elAeVm1allHbsryVYod9gpjSvZyZwX9QVGAm59yt13x0YnGRpq7lNLXwJSA",
    "token_id": "at-a8RCaYBVVtdDNaqJ"
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}

Now I want to place this into an environment variable so that it can be consumed without being written to disk

export RETRIEVED_TOKEN=$(curl \
  --header "X-Vault-Token: $VAULT_TOKEN" \
  --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
  --request GET \
  $VAULT_ADDR/v1/tfc/creds/gitlab-pipeline \
 | jq -r .data.token )

 echo $RETRIEVED_TOKEN
DQ9Fx0W3EEV1Ow.atlasv1.aeAvKQu6elAeVm1allHbsryVYod9gpjSvZyZwX9QVGAm59yt13x0YnGRpq7lNLXwJSA

Given that I just showed you the value of a working API token, I best rotate this as I dont want any of you scallywag readers to access my org!

Rotate Team Token

Rotating is a Team or Organization token is the method of refreshing. This lazy rotation

curl \
  --header "X-Vault-Token: $VAULT_TOKEN" \
  --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
  --request POST \
  $VAULT_ADDR/v1/tfc/rotate-role/gitlab-pipeline

Now if we are to retrieve it again it will not be the same as the previous value in $RETRIEVED_TOKEN

Access Control

What’s super about this is that based on my authentication method I can control who accesses these engines. I would want to ensure whatever role I had for my authentication method included the following for the consumer.

path "tfc/creds/burkey-user" {
  capabilities = [ "read" ]
}
path "tfc/creds/gitlab-pipeline" {
  capabilities = [ "read", "update" ]
}
path "tfc/creds/burkey-org" {
  capabilities = [ "read", "update" ]
}

What this role does is that for this explicit path allow the following actions to be performed. I chose not to do a tfc/creds/* wildcard as I wanted a specific set of child paths to be accessible. This was burkey-user, gitlab-pipeline, and burkey-org. If there was a grant-org then I would need a path configured I would need something to explicitly grant me permission! In this case it is read and write to allow read to retrieve a token and write for the rotation.

An Administrative policy would need additions that pertained to the following:

# Base permissions to actually mount the engine
path "sys/mounts/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# permissions to create the tfc engine
path "tfc/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

These should be used in conjunction with other permissions that allow lease lifecycling.

Summary

This Vault secrets engine allows creation of specific user, team, or organization API tokens. This means that a higher order system such as a pipeline that wants to interact with a commercial offering of Terraform can generate an on demand API token. The role specific within Vault ensures it is managed accordingly to a specific TTL if a user token, and lazily rotated if an organization or team token.

Intro#

Token#

Demo time#

Administration Workflow#

Preparation - Terraform#

Create the secrets engine#

Create the Roles#

CLI Secret Retrieval#

API Secret Retrieval#

Access Control#

Summary#