Dynamic credentials for Kubernetes workloads with Vault and VSO

43min
|
Enterprise
Vault
Interactive

Operating system

The Vault Secrets Operator (VSO) is a Kubernetes Operator that allows teams to centrally manage secrets in Vault and synchronize them with native Kubernetes Secrets.

Scenario

HashiCups, the world's finest coffee roaster, has recently acquired International Brewing Machines (IBM) who makes smart coffee roasters. The teams at HashiCups and IBM need to make the app compliant with HashiCups security standards.

HashiCups recently implemented HashiCorp Vault as part of their security lifecycle management processes. HashiCups requires applications to use short-lived, time bound credentials to limit potential exposure of credentials.

The application is a Kubernetes native application which consumes Kubernetes native secrets. These secrets are manually provisioned, and do not expire.

Alice from the HashiCups architect team meets with Watson from the IBM development team to figure out how they can support the app.

Challenge

The app runs on Kubernetes and retrieves secrets directly from manually provisioned Kubernetes Secrets. The Kubernetes Secret has a username and password for the app to connect to a PostgreSQL database.

The team at HashiCups has the following requirements:

Replace static, long lived credentials with short-lived, time bound credentials credentials.
Allow teams to securely store API keys, passwords, certificates, etc.
Onboard the app quickly, as the teams do not have time to refactor the app.

Solution

To meet the requirement of on-boarding the app quickly, Alice proposes using the Vault Secrets Operator. Alice and Watson decide to configure a static secret in the Vault K/V v2 secrets engine as a proof-of-concept for deploying the VSO. After a successful test with static secrets, they will configure Vault to generate dynamic credentials for PostgreSQL and configure VSO to work with dynamic secrets.

What is the Vault Secrets Operator

Vault provides a complete solution for modern secrets management. It consolidates identity brokering, secrets management, dynamic secrets, rotation, and security policy compliance in one platform. By coupling Vault’s secret management feature set with a Kubernetes Secrets, developers do not have to refactor their applications and can continue to reference Kubernetes Secrets while security and platform teams can manage the lifecycle of secrets through Vault.

How the Vault Secrets Operator works

As a controller, VSO reconciles the current state of secrets defined in the cluster to the desired state specified by custom resources using standard Kubernetes declarative patterns.

You deploy The Vault Secrets Operator using Helm or Kustomize.

The Vault Secrets Operator requires the following Kubernetes permissions:

Object	Permission	Reason
Secret	create, read, update, delete, watch	Sync operations, Vault auth
ServiceAccount	read, token creation	Vault auth
Deployment	read, update, watch	Post secret rotation actions

Prerequisites

This lab was tested on macOS 14.6.1 and Windows 10 with PowerShell Core 7.4.5.

Vault binary installed and configured in your system PATH.
Vault Enterprise license
Docker installed and running
minikube
kubectl
Helm
ngrok installed and configured with an auth token
jq
base64
k9s CLI used for lab support and troubleshooting

Set up the lab

To complete this lab, you will deploy the following:

Kubernetes cluster using minikube.
Vault cluster external to the Kubernetes cluster configured to authenticate with the Kubernetes API.
PostgreSQL with Vault configured to create on-demand PostgreSQL roles.
If you are unable to install the required tools to complete this lab, you can follow this embeded lab for a similar experience.
Launch Terminal
This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Open a terminal and create a working directory to store files created in the lab.
```
$ mkdir -p ~/HC_LAB
```
Change to the the lab directory.
```
$ cd ~/HC_LAB
```
Leave this terminal open for the remainder of the lab.

Start and configure Kubernetes

You will use minikube, a CLI tool that provisions and manages the lifecycle of single-node Kubernetes cluster, to set up a Kubernetes cluster on your system.

Start a Kubernetes cluster.

$ minikube start

😄  minikube v1.25.2 on Darwin 12.3
✨  Automatically selected the docker driver. Other choices: hyperkit, virtualbox, ssh
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=8100MB) ...
🐳  Preparing Kubernetes v1.23.3 on Docker 20.10.12 ...
    ▪ kubelet.housekeeping-interval=5m
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

The initialization process takes several minutes as it retrieves any necessary dependencies and executes various container images.

Verify the status of the Kubernetes cluster.

$ minikube status

minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

Open a new terminal and start a proxy to expose the Kubernetes API.

$ kubectl proxy --disable-filter=true
W0822 12:57:23.605710   10659 proxy.go:177] Request filter disabled, your proxy is vulnerable to XSRF attacks, please be cautious
Starting to serve on 127.0.0.1:8001

This command exposes the Kubernetes API to your local machine. This terminal must remain open during the lab.

Open another new terminal, and create a tunnel to the proxy listening on port 8001.

Warning

ngrok is used to mimic connectivity of complex network configurations to expose the Kubernetes API. Using --scheme=http exposes the API without encryption to avoid TLS certificate errors.

For production workloads, connect to a trusted network with valid TLS certificates.

$ ngrok http --scheme=http 127.0.0.1:8001

Example output:

ngrok                                                                                                                                                              (Ctrl+C to quit)

Session Status                online
Account                       username (Plan: Free)
Update                        update available (version 3.x.x, Ctrl-U to update)
Version                       3.x.x
Region                        United States (us)
Latency                       32.791235ms
Web Interface                 http://127.0.0.1:4040
Forwarding                    http://d12b-34-567-89-10.ngrok.io -> 127.0.0.1:8001

Connections                   ttl     opn     rt1     rt5     p50     p90
                              0       0       0.00    0.00    0.00    0.00

This terminal must remain open during the lab.

ngrok is used for simplicity in exposing the Kubernetes API from minikube which is meant to serve as a local environment. In a production Kubernetes cluster, you might use a load balancer, or managed services such as AKS or EKS expose the API by default.

Copy the forwarding address and return to the terminal where you started minikube.
Export an environment variable for the Kubernetes API address using the ngrok forwarding address. This address will be unique to your lab.
```
$ export K8S_URL=<ngrok-address>
```
The Kubernetes cluster is now accessible by an external Vault cluster.

Start Vault

Export an environment variable with a valid Vault Enterprise license.
```
$ export VAULT_LICENSE=<your-license-key>
```

Start Vault Enterprise in a container. Vault will operate outside the Kubernetes cluster and provides a similar experience to using HCP Vault Dedicated or HCP Vault Secrets with VSO.

$ docker run --name vault-enterprise \
     --cap-add=IPC_LOCK \
     --env VAULT_LICENSE=$(echo $VAULT_LICENSE) \
     --env VAULT_DEV_ROOT_TOKEN_ID=root \
     --env VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200 \
     --publish 8200:8200 \
     --detach \
     --rm \
     hashicorp/vault-enterprise

Example output:

Unable to find image 'hashicorp/vault-enterprise:latest' locally
latest: Pulling from hashicorp/vault-enterprise
6e5015e02423: Pull complete
6be2c75e5421: Pull complete
Digest: sha256:c7e266b84e44580d1f5496c91c23e051f96c75a8774de4b413a563119768c50f
Status: Downloaded newer image for hashicorp/vault-enterprise:latest
5cb93ad2e2e9737761c0903966a0ccc8fd6a835604b30b63a2a23babc8332f59

The Vault dev server listens on all addresses using port 8200. The server is initialized and unsealed.

Insecure operation

Do not run a Vault dev server in production. This approach starts a Vault server with an in-memory database and runs in an insecure way.

Verify the Vault Enterprise container is in a STATUS is Up.

$ docker ps -f name=vault-enterprise

CONTAINER ID   IMAGE                        COMMAND                  CREATED         STATUS         PORTS                    NAMES
5cb93ad2e2e9   hashicorp/vault-enterprise   "docker-entrypoint.s…"   4 minutes ago   Up 4 minutes   0.0.0.0:8200->8200/tcp   vault-enterprise

Export an environment variable for the vault CLI to address the Vault server.
```
$ export VAULT_ADDR=http://127.0.0.1:8200
```
Export an environment variable for the vault CLI to authenticate with the Vault server.
```
$ export VAULT_TOKEN=root
```
Note
For these tasks, you can use Vault's root token. However, it is recommended that root tokens are only used for enough initial setup or in emergencies. As a best practice, use an authentication method or token that meets the policy requirements.

Verify Vault Enterprise is running and unsealed.

$ vault status
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         1.17.6+ent
Build Date      2024-08-06T14:46:20Z
Storage Type    inmem
Cluster Name    vault-cluster-77c01606
Cluster ID      a9a798e7-15ee-ef5d-e62b-72a893480665
HA Enabled      false

The Vault cluster is now ready.

Install the Vault Secrets Operator

Install and update the HashiCorp Helm repository.

$ helm repo add hashicorp https://helm.releases.hashicorp.com \
    && helm repo update

Example output:

"hashicorp" already exists with the same configuration, skipping
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "hashicorp" chart repository
Update Complete. ⎈Happy Helming!⎈

Install the Vault Secrets Operator in a new Kubernetes namespace.
```
$ helm install vault-secrets-operator hashicorp/vault-secrets-operator \
    --namespace vault-secrets-operator \
    --create-namespace
```
Example output:
```
NAME: vault-secrets-operator
LAST DEPLOYED: Thu Jan 25 11:54:37 2024
NAMESPACE: vault-secrets-operator
STATUS: deployed
REVISION: 1
```
In addition to installing the Vault Secrets Operator controller, the installation includes the supported custom resource definitions used to create Kubernetes resources to support the Vault Secrets Operator.

Create a connection to Vault

VSO uses the VaultConnection as a reference to connect to the external Vault cluster.

Create a Kubernetes namespace for Vault resources.
```
$ kubectl create namespace vault
```
A good practice is to use Kubernetes namespaces as a logical security boundary for different resources.
The vault Kubernetes namespace stores the connection to Vault, and a service account used by Vault to authenticate with the Kubernetes API.
Connect to minikube and export an environment variable with the address for the external Vault container.
```
$ K8S_TO_VAULT=$(minikube ssh "dig +short host.docker.internal" | tr -d '\r')
```
In a production environment, this would be the address of your Vault cluster accessible by the Kubernetes API.

Create a connection to Vault.

$ kubectl create -f - <<EOF
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
  namespace: vault
  name: vault-connection
spec:
  # address to the Vault server.
  address: http://$K8S_TO_VAULT:8200
---
EOF

Example output:

vaultconnection.secrets.hashicorp.com/vault-connection created

Verify the configuration.

$ kubectl describe --namespace vault vaultconnection.secrets.hashicorp.com/vault-connection

Name:         vault-connection
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  secrets.hashicorp.com/v1beta1
Kind:         VaultConnection
Metadata:
  Creation Timestamp:  2024-01-25T16:58:47Z
  Finalizers:
    vaultconnection.secrets.hashicorp.com/finalizer
  Generation:        1
  Resource Version:  3720
  UID:               60500026-9195-4e82-bb6b-9d255c1cd8dc
Spec:
  Address:          http://192.168.65.254:8200
  Skip TLS Verify:  false
Status:
  Valid:  true
Events:
  Type    Reason    Age   From             Message
  ----    ------    ----  ----             -------
  Normal  Accepted  107s  VaultConnection  VaultConnection accepted

Create a service account for Vault

Create a Kubernetes service account named vault-to-k8s-sa with a service account token. Vault authenticates with the Kubernetes API using this token.

$ kubectl create -f - <<EOF
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-to-k8s-sa
  namespace: vault
---
apiVersion: v1
kind: Secret
metadata:
  name: vault-to-k8s-sa-secret
  namespace: vault
  annotations:
    kubernetes.io/service-account.name: vault-to-k8s-sa
type: kubernetes.io/service-account-token
---
EOF

Example output:

serviceaccount/vault-to-k8s-sa created
secret/vault-to-k8s-sa-secret created

Create a Kubernetes role for the vault-to-k8s-sa service account to allow access to the Kubernetes API.

$ kubectl create -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-to-k8s-sa-tokenreview-role
  namespace: vault
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: vault-to-k8s-sa
    namespace: vault
EOF

Example output:

clusterrolebinding.rbac.authorization.k8s.io/vault-to-k8s-sa-tokenreview-role created

Retrieve the vault-auth-secret secret and store it as an environment variable.

$ VAULTSA_SECRET=$(kubectl get secret --namespace vault vault-to-k8s-sa-secret --output json | jq -r '.data') \
    && echo $VAULTSA_SECRET

Example output:

{
  "ca.crt": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURC..<snip>..lwNGN6cmFpb0E9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==",
  "namespace": "ZGVmYXVsdA==",
  "token": "ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNkl..<snip>..TWR3T0kwRVlxcE0zZGJ1d3JMUW5vMjNhSnJWaU5SaEp3"
}

The secret includes the Kubernetes public key ca.crt and the token as base64 encoded strings.

Decode the ca.crt certificate and store it as an environment variable.

$ K8S_CA_CRT=$(echo $VAULTSA_SECRET | jq -r '."ca.crt"' | base64 -d) && echo $K8S_CA_CRT

Example output:

-----BEGIN CERTIFICATE-----
MIIDBjCCAe6gAwIBAgIBATANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwptaW5p
a3ViZUNBMB4XDTIyMDQyNjE0MDE0NVoXDTMyMDQyNDE0MDE0NVowFTETMBEGA1UE
...snip...
tIS6p2TV6cW+Fa6VFvRYcjb4hfPovAB5gjhm36YravcjgP9rt+h0rFbQEc5oTtMa
hThaYp4czraioA==
-----END CERTIFICATE-----

Decode the token and store it as an environment variable.

$ VAULTSA_TOKEN=$(echo $VAULTSA_SECRET | jq -r '.token' | base64 -d) && echo $VAULTSA_TOKEN

Example output:

eyJhbGciOiJSUzI1NiIsImtpZCI6Im1xay1Mb3ZRWndUNXowV3hLZnVsdERfTXZYU1hGZXhaNmRMS29NcDhVRU0ifQ.eyJpc3MiOiJrd
WJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia
...snip...
W4KlKrAHUxmWfkmqn-TOg6adqM6gpZ86t3hpNKLieui4k7jIn6MRfvJasuAV7POJ9ZrdKMYWnOE4Hob7H-_Jck0IVBOEpSCd4YIyLU5e
cxOIm9gYnyz7W0P2D4EnWonqPzpAn3YGi5wilRMQ6f6thN8KDz__zYn0Bqx4ioSQ

You have collected the necessary information to configure the Vault Kubernetes auth method.

Enable the Kubernetes auth method

Enable the Kubernetes auth method.

$ vault auth enable kubernetes

Success! Enabled kubernetes auth method at: kubernetes/

Configure the Kubernetes auth method to connect to the Kubernetes API using the vault-to-k8s-sa service account token.
```
$ vault write auth/kubernetes/config \
 token_reviewer_jwt=$VAULTSA_TOKEN \
 kubernetes_host=$K8S_URL \
 kubernetes_ca_cert=$K8S_CA_CRT
```
Example output:
```
Success! Data written to: auth/kubernetes/config
```
The Kubernetes auth method is configured and ready to proceed with the lab.

Enable KV secrets engine

Configure the KV secrets engine to verify the Vault Secrets Operator is working with a basic static secret. The KV secrets engine is useful for storing API keys, and credentials that can not be automatically generated.

Enable the KV secrets engine.
```
$ vault secrets enable -version=2 -path=static kv
Success! Enabled the kv secrets engine at: static/
```
Alice and Watson use the KV secrets engine to validate the IBM app can get static secrets managed by Vault and synchronized to a Kubernetes secret.

Create a secret at path static/exampleapp/creds with a username and password.

$ vault kv put static/exampleapp/creds \
   username='jalbertson' \
   password='worstpasswordever'

Example output:

======== Secret Path ========
static/data/exampleapp/creds

======= Metadata =======
Key                Value
---                -----
created_time       2024-08-22T16:49:33.977869448Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Create a Vault policy that permits read access to static/exampleapp/creds.

$ vault policy write exampleapp-kv-read - << EOF
path "static/data/exampleapp/creds" {
  capabilities = ["read"]
}
EOF

Example output:

Success! Uploaded policy: exampleapp-kv-read

The KV secrets engine and policy is ready to proceed with the lab.

Sync static secrets

Before the Vault Secrets Operator can sync secrets from Vault to Kubernetes Secrets, it needs a Vault role to authenticate with. The policy attached to the role permits access to the KV secrets engine created in the previous section.

Create a Vault role and include the exampleapp-kv-read Vault policy for the Kubernetes service account vso-static-exampleapp-sa. This service account allows VSO to create secrets in the Kubernetes namespace static-exampleapp.
```
$ vault write auth/kubernetes/role/vault-role-static-exampleapp \
bound_service_account_names=vso-static-exampleapp-sa \
bound_service_account_namespaces=static-exampleapp \
policies=exampleapp-kv-read \
ttl=1h
```
Example output:
```
Success! Data written to: auth/kubernetes/role/vault-role-static-exampleapp
```
The bound_service_account_namespaces parameter is a list of Kubernetes namespaces allowed to call the Vault vault-role-static-exampleapp role.
The bound_service_account_names parameter is a list of Kubernetes service accounts allowed to call the Vault vault-role-static-exampleapp role.
You will now create a Kubernetes namespace to deploy the app. This namespace will also include a Kubernetes service account. This service account must exist in the Kubernetes namespace where you want VSO to write the secret to.

Create a Kubernetes namespace to create the secret in and deploy the app.

$ kubectl create namespace static-exampleapp
namespace/static-exampleapp created

Create a Kubernetes service account for VSO to use to create a secret in the static-exampleapp Kubernetes namespace.
```
$ kubectl create -f - <<EOF
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vso-static-exampleapp-sa
  namespace: static-exampleapp
---
EOF
```
Example output:
```
serviceaccount/vso-static-exampleapp-sa created
```
Service accounts can perform all operations in the namespace in which it is created, such as creating secrets.

Configure authentication for the Vault Secrets Operator using the vso-static-exampleapp-sa Kubernetes service account.

$ kubectl create -f - <<EOF
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: vault-auth-vso-static-exampleapp
  namespace: static-exampleapp
spec:
  vaultConnectionRef: vault/vault-connection
  method: kubernetes
  mount: kubernetes
  allowedNamespaces:
    - static-exampleapp
  kubernetes:
    role: vault-role-static-exampleapp
    serviceAccount: vso-static-exampleapp-sa
---
EOF

Example output:

vaultauth.secrets.hashicorp.com/vault-auth-vso-static-exampleapp created

Configure the Vault Secrets Operator to read from the secret KV v2 mount at the exampleapp/creds path.

$ kubectl create -f - <<EOF
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: vault-static-secret
  namespace: static-exampleapp
spec:
  vaultAuthRef: vault-auth-vso-static-exampleapp
  mount: static
  type: kv-v2
  path:  exampleapp/creds
# version: 2
  refreshAfter: 300s
  destination:
    create: true
    name: vso-static-creds-from-vault
---
EOF

Example output:

vaultstaticsecret.secrets.hashicorp.com/vault-static-secret created

Verify the Vault secret is available as a native Kubernetes Secret.

$ kubectl get secrets --namespace static-exampleapp

NAME                              TYPE                                  DATA   AGE
vso-static-creds-from-vault       Opaque                                3      15s

VSO creates the secret vso-static-creds-from-vault and bases the name on the name parameter provided in the VaultStaticSecret configuration.

Read the Kubernetes Secret value and decode the base64 encoded strings.

$ kubectl get secret --namespace static-exampleapp vso-static-creds-from-vault -o json | jq ".data | map_values(@base64d)"

{
  "_raw": "{\"data\":{\"password\":\"worstpasswordever\",\"username\":\"jalbertson\"},\"metadata\":{\"created_time\":\"2024-01-25T15:48:31.871429498Z\",\"custom_metadata\":null,\"deletion_time\":\"\",\"destroyed\":false,\"version\":1}}",
  "password": "worstpasswordever",
  "username": "jalbertson"
}

Applications can now use the Kubernetes Secret by injecting it through a volume mount or an environment variable.

Create a deployment for the app that reads the Kubernetes Secret and creates an environment variable in the pod.

$ tee deployment.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hashicups-exampleapp
  namespace: static-exampleapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hashicups-exampleapp
  template:
    metadata:
      labels:
        app: hashicups-exampleapp
    spec:
      containers:
      - name: test
        image: jfrappier/dynamic-exampleapp:latest
        ports:
        - containerPort: 5000
        env:
        - name: DB_USER
          valueFrom:
            secretKeyRef:
              name: vso-static-creds-from-vault
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: vso-static-creds-from-vault
              key: password
        - name: DB_HOST
          value: "your_db_host"
        - name: DB_PORT
          value: "5432"
        - name: DB_NAME
          value: "your_db_name"
EOF

The app references the Kubernetes Secret vso-static-creds-from-vault and injects an environment variable to the running pod.

Deploy the hashicups-exampleapp using the deployment.yaml file.

$ kubectl apply --namespace static-exampleapp -f deployment.yaml

Verify the pod is in a running state. It may take a few minutes to download the image and start the pod.

$ kubectl get pods --namespace static-exampleapp
NAME                                    READY   STATUS    RESTARTS   AGE
hashicups-exampleapp-645f8b5867-8qll7   1/1     Running   0          4m

Access the applications secret endpoint to display the Kubernetes Secret created by VSO.
```
$ kubectl exec --namespace static-exampleapp --stdin=true \
    $(kubectl get pods --namespace static-exampleapp -l app=hashicups-exampleapp -o name) \
    -- curl http://127.0.0.1:5000/secret --silent
```
Example output:
```
{"password":"WorstPasswordEver","username":"jalbertson"}
```
The app was able to read the Kubernetes Secret created by VSO. The deployment injected the secrets as environment variables, however you can also can retrieve secrets through the Vault Injector service via annotations, or secrets mounted on ephemeral volumes.
Delete the exampleapp deployment.
```
$ kubectl delete --namespace static-exampleapp -f deployment.yaml
```
This step is done only for lab purposes to conserve resources on your local machine.

Sync dynamic secrets

Alice and Watson have successfully tested the app with Vault and VSO. VSO creates the Kubernetes Secret from a Vault managed secret, and the app works without having to develop new features to read the Vault managed secret.

The Vault database secrets engine connects to a supported RDBMS and manages the creation and revocation of credentials. When a credentials TTL is reached, Vault revokes the credentials.

Alice and Watson will test Vault's database secrets engine, which creates dynamic, just-in-time credentials. These credentials are short lived, limiting potential exposure of the credentials.

Start and configure PostgreSQL

Start PostgreSQL.

$ docker run \
     --detach \
     --name learn-postgres \
     -e POSTGRES_USER=root \
     -e POSTGRES_PASSWORD=rootpassword \
     -p 5432:5432 \
     --rm \
     postgres

Example output:

Unable to find image 'postgres:latest' locally
latest: Pulling from library/postgres
e4fff0779e6d: Already exists
3dd23fa89c28: Pull complete
...snip...
133de8acf4aa: Pull complete
Digest: sha256:c62fdb7fd6f519ef425c54760894c74e8d0cb04fbf4f7d3d79aafd86bae24edd
Status: Downloaded newer image for postgres:latest
ebeb6f879088f172b6993dbcc67e6d46c9158d01e1bbae7c7dd17a9dd3ae45ca

Connect to minikube and export an environment variable with the address for the PostgreSQL container.
```
$ K8S_TO_POSTGRESQL=$(minikube ssh "dig +short host.docker.internal" | tr -d '\r')
```
In a production environment, this would be the address of your PostgreSQL server accessible by Vault.
Enable the database secrets engine.
```
$ vault secrets enable database
```

Configure the database secrets engine to use the postgresql-database-plugin, and the PostgreSQL root credentials.

$ vault write database/config/postgresql \
     plugin_name=postgresql-database-plugin \
     connection_url="postgresql://{{username}}:{{password}}@$K8S_TO_POSTGRESQL/postgres?sslmode=disable" \
     allowed_roles=vault-role-dynamic-exampleapp \
     username="root" \
     password="rootpassword"

Configure a Vault role named vault-role-dynamic-exampleapp that includes a SQL statement to create the PostgreSQL role.
```
$ vault write database/roles/vault-role-dynamic-exampleapp \
     db_name="postgresql" \
     creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
         GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
     default_ttl="1h" \
     max_ttl="24h"
```
In production environments, a good practice is to include a valid revocation statement which are valid for the database you've configured. If you do not specify statements appropriate to creating, revoking, or rotating users, Vault inserts generic statements which can be unsuitable for your deployment.

Test the Vault role and generate PostgreSQL credentials.

$ vault read database/creds/vault-role-dynamic-exampleapp

Example output:

Key                Value
---                -----
lease_id           database/creds/vault-role-dynamic-exampleapp/JzEIFnrPGDWrxvxyvAHeLTyZ
lease_duration     1h
lease_renewable    true
password           aNUEd-agvqnclxpkAxG4
username           v-token-vault-ro-DBrelKpqZTyhuEI2lXYh-1724964233

Vault uses the creation_statements from the vault-role-dynamic-exampleapp Vault role to create a new PostgreSQL username and password.

Create and sync dynamic secrets

With the database secrets engine configured Alice and Watson will configure VSO to create and sync the new credentials to a Kubernetes Secret.

Create a policy that allows access to the database secrets engine.

$ vault policy write exampleapp-database-read - <<EOF
path "database/creds/vault-role-dynamic-exampleapp" {
  capabilities = [ "read" ]
}
EOF

Example output:

Success! Uploaded policy: exampleapp-database-read

Create a Vault role for the app with the exampleapp-database-read policy attached.

$ vault write auth/kubernetes/role/vault-role-dynamic-exampleapp \
bound_service_account_names=vso-dynamic-exampleapp-sa \
bound_service_account_namespaces=dynamic-exampleapp \
policies=exampleapp-database-read \
ttl=1h

Example output:

Success! Data written to: auth/kubernetes/role/vault-role-dynamic-exampleapp

A good practice is to create separate Vault roles for each Kubernetes namespace.

Create a Kubernetes namespace to store the dynamic secrets and run the app.
```
$ kubectl create namespace dynamic-exampleapp
namespace/dynamic-exampleapp created
```
A new Kubernetes namespace is used to show the steps specific to configuring VSO for dynamic secrets. Certain Kubernetes custom resources such as the VaultConnection resource can be shared across different namespaces.

Create a Kubernetes service account for VSO to use to create a dynamic secret in the dynamic-exampleapp namespace.

$ kubectl create -f - <<EOF
---
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: dynamic-exampleapp
  name: vso-dynamic-exampleapp-sa
---
EOF

Example output:

serviceaccount/vso-dynamic-exampleapp-sa created

Configure authentication for the Vault Secrets Operator using the Kubernetes vso-static-exampleapp-sa service account and the vault-role-dynamic-exampleapp Vault role.

$ kubectl create -f - <<EOF
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: vault-auth-vso-dynamic-exampleapp
  namespace: dynamic-exampleapp
spec:
  vaultConnectionRef: vault/vault-connection
  method: kubernetes
  mount: kubernetes
  allowedNamespaces:
    - dynamic-exampleapp
  kubernetes:
    role: vault-role-dynamic-exampleapp
    serviceAccount: vso-dynamic-exampleapp-sa
---
EOF

Example output:

vaultauth.secrets.hashicorp.com/vault-auth-vso-dynamic-exampleapp created

The connection resource vaultConnectionRef is reused from earlier in the lab.

Configure the Vault Secrets Operator to read from the database mount using the vault-role-dynamic-exampleapp role.
```
$ kubectl create -f - <<EOF
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  name: vso-db-demo-create
  namespace: dynamic-exampleapp
spec:
  vaultAuthRef: vault-auth-vso-dynamic-exampleapp
  mount: database
  path: creds/vault-role-dynamic-exampleapp
  destination:
    create: true
    name: vso-dynamic-creds-from-vault
  rolloutRestartTargets:
  - kind: Deployment
    name: hashicups-exampleapp
---
EOF
```
Example output:
```
vaultdynamicsecret.secrets.hashicorp.com/vso-db-demo-create created
```
VSO will create a Kubernetes Secret named vso-dynamic-creds-from-vault using the secrets engine mounted at the path database using the vault-role-dynamic-exampleapp Vault role.
The rolloutRestartTargets can be configured whenever the application(s) consuming the Vault secret does not support dynamically reloading a rotated secret. In that case one, or more RolloutRestartTargets can be configured. VSO will trigger a "rollout-restart" for each target whenever the Vault secret changes between reconciliation events.

Verify the Vault database credentials are available as a native Kubernetes secret.

$ kubectl get secrets --namespace dynamic-exampleapp

Example output:

NAME                           TYPE     DATA   AGE
vso-dynamic-creds-from-vault   Opaque   3      10s

Read the Kubernetes Secret value and decode the base64 encoded strings.

$ kubectl get secret --namespace dynamic-exampleapp vso-dynamic-creds-from-vault -o json | jq ".data | map_values(@base64d)"
{
  "_raw": "{\"password\":\"CWZODcdw-Di97dVWzRgd\",\"username\":\"v-kubernet-vault-ro-W4Y577NGFzE7gJgdqXuW-1726076758\"}",
  "password": "CWZODcdw-Di97dVWzRgd",
  "username": "v-kubernet-vault-ro-W4Y577NGFzE7gJgdqXuW-1726076758"
}

Create a deployment for the app that reads the Kubernetes Secret.

$ tee deployment.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hashicups-exampleapp
  namespace: dynamic-exampleapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hashicups-exampleapp
  template:
    metadata:
      labels:
        app: hashicups-exampleapp
    spec:
      containers:
      - name: test
        image: jfrappier/dynamic-exampleapp:latest
        ports:
        - containerPort: 5000
        env:
        - name: DB_USER
          valueFrom:
            secretKeyRef:
              name: vso-dynamic-creds-from-vault
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: vso-dynamic-creds-from-vault
              key: password
        - name: DB_HOST
          value: "$K8S_TO_POSTGRESQL"
        - name: DB_PORT
          value: "5432"
        - name: DB_NAME
          value: "postgres"
EOF

Deploy the hashicups-exampleapp using the deployment.yaml file.

$ kubectl apply --namespace dynamic-exampleapp -f deployment.yaml
deployment.apps/hashicups-exampleapp created

Verify the pod is in a running state.

$ kubectl get pods --namespace dynamic-exampleapp
NAME                                    READY   STATUS    RESTARTS   AGE
hashicups-exampleapp-7cfb565cd5-vkbrx   1/1     Running   0          37s

Access the applications secret endpoint to display the Kubernetes Secret created by VSO.

$ kubectl exec --namespace dynamic-exampleapp --stdin=true \
 $(kubectl get pods --namespace dynamic-exampleapp -l app=hashicups-exampleapp -o name) \
 -- curl http://127.0.0.1:5000/secret --silent

Example output:

{"password":"CWZODcdw-Di97dVWzRgd","username":"v-kubernet-vault-ro-W4Y577NGFzE7gJgdqXuW-1726076758"}

Access the applications status endpoint to verify the pod can connect to the PostgreSQL database.

$ kubectl exec --namespace dynamic-exampleapp --stdin=true \
 $(kubectl get pods --namespace dynamic-exampleapp -l app=hashicups-exampleapp -o name) \
 -- curl http://127.0.0.1:5000/status --silent

Example output:

{"message":"Connected to the database successfully","status":"success"}

The app connected to the PostgreSQL database using the dynamic secrets created by Vault, and synchronized to a Kubernetes Secret by VSO.

Set up persistent client cache

By default, the Vault client cache does not persist. This can lead to VSO not maintaining dynamic secret leases through restarts and upgrades of the Vault Secrets Operator.

The client cache enables seamless upgrades because Vault can track and renew tokens and dynamic secret leases through leadership changes.

Client cache persistence and encryption is not enabled by default because it requires extra configuration and Vault server setup.

Enable the transit secrets engine.

$ vault secrets enable  transit
Success! Enabled the transit secrets engine at: transit/

Create an encryption key.

$ vault write -force transit/keys/vso-client-cache
Key                       Value
---                       -----
allow_plaintext_backup    false
auto_rotate_period        0s
deletion_allowed          false
derived                   false
exportable                false
imported_key              false
keys                      map[1:1726154823]
latest_version            1
min_available_version     0
min_decryption_version    1
min_encryption_version    0
name                      vso-client-cache
supports_decryption       true
supports_derivation       true
supports_encryption       true
supports_signing          false
type                      aes256-gcm96

Create a policy that allows VSO to access the transit secrets engine.

$ vault policy write vso-transit-policy - <<EOF
path "transit/encrypt/vso-client-cache" {
   capabilities = ["create", "update"]
}
path "transit/decrypt/vso-client-cache" {
   capabilities = ["create", "update"]
}
EOF

Example output:

Success! Uploaded policy: vso-transit-policy

Create a role in the Kubernetes auth method used by VSO to authenticate with Vault and attach the vso-transit-policy.

$ vault write auth/kubernetes/role/operator \
  bound_service_account_names=vault-secrets-operator-controller-manager \
  bound_service_account_namespaces=vault-secrets-operator \
  token_period="24h" \
  token_policies=vso-transit-policy

Example output:

Success! Data written to: auth/kubernetes/role/operator

The vault-secrets-operator-controller-manager service account and vault-secrets-operator are the defaults created when deploying VSO.

Create a file to update the VSO Helm deployment.

$ tee update-values.yml <<EOF
controller:
  manager:
    clientCache:
      persistenceModel: direct-encrypted
      storageEncryption:
        enabled: true
        vaultConnectionRef: vault/vault-connection
        keyName: vso-client-cache
        transitMount: transit
        method: kubernetes
        mount: kubernetes
        kubernetes:
          role: operator
          serviceAccount: vault-secrets-operator-controller-manager
EOF

Upgrade the VSO deployment to include the secrets cache.

$ helm upgrade -f update-values.yml vault-secrets-operator hashicorp/vault-secrets-operator \
    --namespace vault-secrets-operator

Example output:

Release "vault-secrets-operator" has been upgraded. Happy Helming!
NAME: vault-secrets-operator
LAST DEPLOYED: Thu Sep 12 11:30:37 2024
NAMESPACE: vault-secrets-operator
STATUS: deployed
REVISION: 2

Check the vault-secrets-operator namespace for the encrypted client cache. Depending on your environment, this may take 30-60 seconds to update.

$ kubectl get secrets -n vault-secrets-operator
sh.helm.release.v1.vault-secrets-operator.v1   helm.sh/release.v1   1      44h
sh.helm.release.v1.vault-secrets-operator.v2   helm.sh/release.v1   1      53s
vso-cc-kubernetes-a1fd22e2726ce4c715a25f       Opaque               2      20s
vso-cc-kubernetes-e086c5f655898e3544b9b8       Opaque               2      20s
vso-cc-storage-hmac-key                        Opaque               1      44h

VSO stores the client cache as a Kubernetes Secret prefixed with vso-cc in the vault-secrets-operator namespace.

In this example, there is a cache for both Kubernetes namespaces VSO is managing secrets for - static-exampleapp and dynamic-exampleapp.

Set up instant updates with Vault Enterprise

Tip

The instant updates option requires Vault Enterprise 1.16.3+ due to the use of Vault event notifications.

There are times when secrets updated in Vault need to be immediately available to applications to avoid downtime. To ensure the timely update of secrets by the Vault Secrets Operator, you can use Vault event notifications.

Instant update supports the Vault KV v2 secrets engine, and the Kubernetes VaultStaticSecret resource type.

Update the Vault exampleapp-kv-read policy to include read access to events, as well as list and subscribe on the exampleapp-kv-read policy.

$ vault policy write exampleapp-kv-read - << EOF
path "static/data/exampleapp/creds" {
  capabilities = ["read", "list", "subscribe"]
  subscribe_event_types = ["*"]
}

path "sys/events/subscribe/kv*" {
  capabilities = ["read"]
}
EOF

Example output:

Success! Uploaded policy: exampleapp-kv-read

Add syncConfig.instantUpdates=true to the vault-static-secret resource in the static-exampleapp namespace.

$ kubectl apply -f - <<EOF
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: vault-static-secret
  namespace: static-exampleapp
spec:
  vaultAuthRef: vault-auth-vso-static-exampleapp
  mount: static
  type: kv-v2
  path:  exampleapp/creds
# version: 2
  refreshAfter: 300s
  destination:
    create: true
    name: vso-static-creds-from-vault
  syncConfig:
    instantUpdates: true
---
EOF

Note

It is okay to proceed if you receive the following error message:

Warning: resource vaultstaticsecrets/vault-static-secret is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.

Example output:

vaultstaticsecret.secrets.hashicorp.com/vault-static-secret configured

Verify the VSO configuration is now watching for events.

$ kubectl describe vaultstaticsecret vault-static-secret --namespace static-exampleapp
...snip...
Events:
Type    Reason               Age   From               Message
----    ------               ----  ----               -------
Normal  SecretRotated        110s  VaultStaticSecret  Secret synced
Normal  EventWatcherStarted  110s  VaultStaticSecret  Started watching events

Update the /static/exampleapp/creds secret in Vault.

$ vault kv put static/exampleapp/creds \
   username='jalbertson' \
   password='bestpasswordever'

Example output:

======== Secret Path ========
static/data/exampleapp/creds

======= Metadata =======
Key                Value
---                -----
created_time       2024-10-11T17:25:40.390838793Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            2

Verify the Kubernetes Secret values match the updated Vault secret value.

$ kubectl get secret --namespace static-exampleapp vso-static-creds-from-vault -o json | jq ".data | map_values(@base64d)"
{
  "_raw": "{\"data\":{\"password\":\"bestpasswordever\",\"username\":\"jalbertson\"},\"metadata\":{\"created_time\":\"2024-09-30T19:17:39.666589844Z\",\"custom_metadata\":null,\"deletion_time\":\"\",\"destroyed\":false,\"version\":2}}",
  "password": "bestpasswordever",
  "username": "jalbertson"
}

Updating the secret value triggered an event in Vault, which initiated updating the Kubernetes Secret.

Summary

In this lab, you deployed the Vault Secrets Operator and configured VSO to work with an external Vault cluster. You then configured VSO to read both static and dynamic secrets and create Kubernetes native secrets in multiple Kubernetes namespaces. Finally, you deployed the application and validated it can read the secrets created by VSO by connecting the application to the PostgreSQL database.

Now that the app is set up in support of the new smart brewing machines, Alice and Watson head to that famous Boston donut shop to enjoy a well deserved iced coffee (Yea, I know, iced coffee in October? It's a Boston thing!)!

Clean up

Stop the Vault Enterprise, and PostgreSQL container.
```
$ docker kill vault-enterprise learn-postgres
```
Stop minikube.
```
$ minikube stop
```
(Optional) Delete the minikube instance.
```
$ minikube delete
```
Return to the terminal running the Kubernetes proxy and type ctrl-c to stop the proxy.
Return to the terminal running the ngrok type ctrl-c to stop ngrok.
Delete the HC_LAB directory and lab content.
```
$ rm -rf ~/HC_LAB
```

Help and reference

The Vault Secrets Operator on Kubernetes

Next Collection

Authentication