Terraform Cloud secrets engine

As adoption of HCP Terraform grows, more organizations are incorporating it into their automated workflows and existing tooling. Interaction with the HCP TerrafromAPI relies on auth tokens generated by the API and used by external systems to automate actions in Terraform Cloud, often as part of an organization’s CI/CD pipelines. Operators are left with the responsibility of tracking which tokens are in-use by their organizations.

The Vault Terraform Cloud secrets engine enables you to generate, manage and revoke credentials for Terraform Cloud and Terraform Enterprise while adhering to best practices of access and control.

In this tutorial, you will enable the secrets engine, configure it to generate credentials, and then manage those credentials.

Prerequisites

To perform the tasks described in this tutorial, you need to have:

• An HCP Vault cluster or a self-hosted Vault Enterprise environment.

• Terraform Cloud account

Policy requirements

Each persona requires a different set of capabilities. These are expressed in policies. If you are not familiar with policies, complete the policies tutorial.

The admin tasks require these capabilities.

# Mount secrets engines
path "sys/mounts/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Configure the Terraform Cloud secrets engine and create roles
path "terraform/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Manage the leases
path "sys/leases/+/terraform/creds/my-user/*" {
  capabilities = [ "create", "read", "update", "delete", "list", "sudo" ]
}

path "sys/leases/+/terraform/creds/my-user" {
  capabilities = [ "create", "read", "update", "delete", "list", "sudo" ]
}

# Write ACL policies
path "sys/policies/acl/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Manage tokens for verification
path "auth/token/create" {
  capabilities = [ "create", "read", "update", "delete", "list", "sudo" ]
}

The apps tasks require these capabilities.

# Get credentials from the terraform secrets engine
path "terraform/creds/my-user" {
  capabilities = [ "read" ]
}

Lab setup

The tutorial requires a Terraform Cloud API key.

Retrieve Terraform Cloud API key

1. Launch a web browser, and enter https://www.terraform.io/cloud.

2. Click Sign In.

3. Enter your username in the Username or email field.

4. Enter your password in the Password field.

5. Select Sign in.

6. Expand your profile, and select User settings.

7. From the side navigation, select Tokens.

8. Select Create an API token.

9. Enter "learn-vault" in the Description field.

10. Select Create API token.

11. Select the copy icon displayed after the token to copy it.

12. In a terminal, export the variable TF_TOKEN to the copied token value.

    $ export TF_TOKEN=<Copied Token>
    

13. Select Done.

    This token enables you to generate tokens for your Terraform Cloud user account. Tokens generated for teams and organizations would be able to generate tokens for their respective scope.

Start Vault

Scenario Introduction

In this tutorial, you will configure the Terraform Cloud secrets engine, create a role to generate tokens and then manage the lifecycle of these tokens.

1. Enable the Terraform Cloud secrets engine
2. Configure Terraform Cloud secrets engine
3. Create a role
4. Request Terraform credentials
5. Manage leases

Enable the Terraform Cloud secrets engine

The Terraform Cloud secrets engine generates Terraform Cloud tokens dynamically based on configured roles.

$ vault secrets enable terraform
Success! Enabled the terraform secrets engine at: terraform/

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
       --request POST \
       --data '{"type":"terraform"}' \
       $VAULT_ADDR/v1/sys/mounts/terraform

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request POST \
   --data '{"type":"terraform"}' \
   $VAULT_ADDR/v1/sys/mounts/terraform

The Terraform Cloud secrets engine is enabled.

Configure Terraform Cloud secrets engine

The Terraform Cloud secrets engine is configured by default to communicate with Terraform Cloud. The Terraform Cloud API key is set in the configuration to authenticate.

Configure the Terraform Cloud secrets engine to use the TF_TOKEN token.

$ vault write terraform/config token=$TF_TOKEN

$ tee payload.json <<EOF
{
    "token": "$TF_TOKEN"
}
EOF

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data @payload.json \
    $VAULT_ADDR/v1/terraform/config

$ tee payload.json <<EOF
{
    "token": "$TF_TOKEN"
}
EOF

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request POST \
   --data @payload.json \
   $VAULT_ADDR/v1/terraform/config

Create a role

The secrets engine is configured with the credentials that you provided it. These credentials are used through roles that you define for each secrets engine. A role is a logical name within Vault that maps to Terraform Cloud credentials. These roles are defined for an organization, a team, or a user.

Request the Terraform user ID for the account associated with the TF_TOKEN.

$ curl -s \
    --header "Authorization: Bearer $TF_TOKEN" \
    --header "Content-Type: application/vnd.api+json" \
    --request GET \
    https://app.terraform.io/api/v2/account/details | jq -r ".data.id"

NOTE: This example uses jq to process the JSON output for readability.

This unique ID is required to generate credentials for this user.

Create a variable to store the user ID.

$ USER_ID=$(!!)

Create a role named my-user with the USER_ID and a time-to-live of 2 minutes.

$ vault write terraform/role/my-user user_id=$USER_ID ttl=2m
Success! Data written to: terraform/role/my-user

This role is ready to generate credentials for this user that stay valid for two minutes.

Request Terraform credentials

The applications or users that require the Terraform credentials read them from the secrets engine's my-user role.

Read credentials from the my-user role.

$ vault read terraform/creds/my-user
Key                Value
---                -----
lease_id           terraform/creds/my-user/00hjA2DN0deht4z3Lp7oniEg
lease_duration     2m
lease_renewable    true
token              VgySVQ9kfyR6Yg.atlasv1.ogU7VVMAH8kH7A8chrCIOg64yxUM0l5iVldHncLOhWqUInh0aSure5uYYgRjyQBM3YA
token_id           at-adFBZFRExz9KiZqV

The Terraform credentials are displayed as the token value. The token_id represents its unique identifier that Terraform Cloud uses to maintain that ID.

Create a variable to store the token created from the my-user role.

$ CREATED_TF_TOKEN=$(vault read -format=json terraform/creds/my-user | jq -r ".data.token")

Request all the authentication token IDs for the user account authenticating with the CREATED_TF_TOKEN token.

$ curl \
  --header "Authorization: Bearer $CREATED_TF_TOKEN" \
  --header "Content-Type: application/vnd.api+json" \
  --request GET \
  https://app.terraform.io/api/v2/users/$USER_ID/authentication-tokens | jq -r ".data[].id"

The results of the request authenticates with the new token and returns the list of all the token ids generated.

Manage leases

The credentials are managed by the lease ID and remain valid for the lease duration (TTL) or until revoked. Once revoked the credentials are no longer valid.

List the existing leases.

$ vault list sys/leases/lookup/terraform/creds/my-user

Keys
----
IQKUMCTg3M5QTRZ0abmLKjTX

All valid leases for Terraform credentials are displayed.

Create a variable that stores the first lease ID.

$ LEASE_ID=$(vault list -format=json sys/leases/lookup/terraform/creds/my-user | jq -r ".[0]")

Renew the lease for the database credential by passing its lease ID.

$ vault lease renew terraform/creds/my-user/$LEASE_ID

Key                Value
---                -----
lease_id           terraform/creds/my-user/aKziIqZM5GAm39swiJWEkp2g
lease_duration     2m
lease_renewable    true

The TTL of the renewed lease is set to 2m.

Revoke the lease without waiting for its expiration.

$ vault lease revoke terraform/creds/my-user/$LEASE_ID
All revocation operations queued successfully!

List the existing leases.

$ vault list sys/leases/lookup/terraform/creds/my-user
No value found at sys/leases/lookup/terraform/creds/my-user/

The lease is no longer valid and is not displayed.

Read new credentials from the my-user role.

$ vault read terraform/creds/my-user
Key                Value
---                -----
lease_id           terraform/creds/my-user/Y4CD97WFsEL13d4AMnEgOUlY
lease_duration     1m
lease_renewable    true
token              ndklHRyo22tSvQ.atlasv1.mmrM5mEOxZFg3yPtvdJiviWKgyjL2wQOIuVhvqVtfiszhSOwezypiHv7FydUMs7eVSE
token_id           at-c13pERGbFXRiGd56

All leases associated with a path may be removed.

Revoke all the leases with the prefix terraform/creds/my-user.

$ vault lease revoke -prefix terraform/creds/my-user

The prefix flag matches all valid leases with the path prefix of terraform/creds/my-user.

List the existing leases.

$ vault list sys/leases/lookup/terraform/creds/my-user
No value found at sys/leases/lookup/terraform/creds/my-user

All the leases with this path as a prefix have been revoked.

Next Step

The Terraform Cloud tokens that you generated are able to perform all the abilities of the account that created them. Learn more about Terraform Cloud with the Terraform Cloud - Get Started tutorials.

The Terraform Cloud secrets engine is configured to work with your user account. Learn how to configure it for organizations and teams by by reading the Terraform secrets engine documentation.