HashiCorp Vault Secrets Management: Architecture, Dynamic Credentials, and Kubernetes Integration

Arxentra TeamPublished March 5, 202619 min read

HashiCorp Vaultsecrets managementKubernetesdynamic credentialsPKIsecurity

Secrets sprawl is the leading cause of cloud credential compromise. The 2025 Verizon Data Breach Investigations Report found that 68% of breaches involved credential abuse — and most originated from hardcoded secrets in source code, long-lived static tokens in CI/CD systems, or shared service account passwords rotated less than once per year. HashiCorp Vault eliminates all three failure modes through centralized secrets management, dynamic credential issuance, and fine-grained audit logging.

This guide covers Vault architecture end-to-end: seal/unseal mechanics, storage backends, dynamic database credentials, PKI for TLS automation, transit encryption as a service, Kubernetes auth integration, and a detailed comparison against AWS Secrets Manager, Azure Key Vault, and GCP Secret Manager.

---

Vault Architecture: Seal, Unseal, and Storage Backends

Vault stores all data encrypted at rest using AES-256-GCM. On startup, Vault is sealed — it holds the encryption key in memory but cannot access its storage until unsealed. This design means a compromised storage backend (S3, Consul, etcd) yields only encrypted ciphertext.

Seal/Unseal Mechanics

The master encryption key is split using Shamir's Secret Sharing. By default, Vault creates 5 key shards and requires 3 (the threshold) to unseal. No single operator can unseal Vault alone.

<h1>Initialize Vault (first-time only)</h1>
vault operator init \
  -key-shares=5 \
  -key-threshold=3 \
  -format=json > vault-init.json

<h1>Store unseal keys in separate secure locations — NEVER in the same system</h1>
<h1>Unseal with 3 of 5 keys</h1>
vault operator unseal $(cat vault-init.json | jq -r ".unseal_keys_b64[0]")
vault operator unseal $(cat vault-init.json | jq -r ".unseal_keys_b64[1]")
vault operator unseal $(cat vault-init.json | jq -r ".unseal_keys_b64[2]")

<h1>Check seal status</h1>
vault status

Auto-Unseal with cloud KMS (production recommendation):

<h1>vault-config.hcl</h1>
seal "awskms" {
  region     = "us-east-1"
  kms_key_id = "arn:aws:kms:us-east-1:123456789:key/mrk-abc123"
}

Storage Backends

Backend

HA Support

Recommended For

Notes

Integrated Raft	Yes (built-in)	Most deployments	Default since Vault 1.4; no external dependency
Consul	Yes	Existing Consul users	Adds operational complexity
S3	No	Dev/test only	No HA; single point of failure
Azure Blob	No	Dev/test only	No HA
etcd	Yes	Kubernetes-native	Requires etcd cluster management

Integrated Raft is the recommended storage backend for new deployments. It provides HA without an external dependency and reduces your operational surface area significantly.

---

Dynamic Database Credentials

Static database passwords are rotated infrequently and shared across services. Vault's database secrets engine issues time-limited credentials on demand — each application gets a unique username and password that expires automatically.

<h1>Enable the database secrets engine</h1>
vault secrets enable database

<h1>Configure PostgreSQL connection</h1>
vault write database/config/my-postgresql-db \
  plugin_name=postgresql-database-plugin \
  allowed_roles="api-service,worker-service" \
  connection_url="postgresql://{{username}}:{{password}}@postgres:5432/appdb" \
  username="vault-admin" \
  password="initial-admin-password" \
  password_authentication="scram-sha-256"

<h1>Create a role that generates short-lived credentials</h1>
vault write database/roles/api-service \
  db_name=my-postgresql-db \
  creation_statements="CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO "{{name}}";" \
  revocation_statements="DROP ROLE IF EXISTS "{{name}}";" \
  default_ttl="1h" \
  max_ttl="24h"

<h1>Application requests credentials at runtime</h1>
vault read database/creds/api-service
<h1>Returns: username=v-api-service-xK9mP2, password=A3f-random-secure, lease_duration=1h</h1>

Dynamic credential flow in an application:

import hvac
import os

def get_db_credentials():
    """Fetch short-lived DB credentials from Vault at startup."""
    client = hvac.Client(
        url=os.environ["VAULT_ADDR"],
        token=os.environ["VAULT_TOKEN"],  # Injected by Vault Agent
    )
    creds = client.secrets.database.generate_credentials(name="api-service")
    return {
        "user": creds["data"]["username"],
        "password": creds["data"]["password"],
        "lease_id": creds["lease_id"],
        "lease_duration": creds["lease_duration"],
    }

---

PKI Engine: Automated TLS Certificate Management

Vault's PKI secrets engine issues X.509 certificates on demand, eliminating the need for manual certificate management and enabling short-lived certificates (24h or less) that reduce your exposure window.

<h1>Enable PKI and configure Root CA</h1>
vault secrets enable pki
vault secrets tune -max-lease-ttl=87600h pki

vault write pki/root/generate/internal \
  common_name="Arxentra Internal Root CA" \
  ttl=87600h

<h1>Enable intermediate CA</h1>
vault secrets enable -path=pki_int pki

<h1>Issue certificates for services</h1>
vault write pki_int/roles/web-service \
  allowed_domains="svc.cluster.local,internal.arxentra.io" \
  allow_subdomains=true \
  max_ttl="72h"

<h1>Request a certificate</h1>
vault write pki_int/issue/web-service \
  common_name="api.svc.cluster.local" \
  ttl="24h"

---

Transit Encryption: Encryption as a Service

The transit secrets engine provides encryption without storing data in Vault. Applications encrypt sensitive fields (PII, payment data, health records) and Vault handles key management, rotation, and convergent encryption.

<h1>Enable transit and create an encryption key</h1>
vault secrets enable transit
vault write -f transit/keys/user-pii-key

<h1>Encrypt data (returns ciphertext, never the key)</h1>
vault write transit/encrypt/user-pii-key \
  plaintext=$(echo -n "[email protected]" | base64)

<h1>Application usage via API</h1>
curl -X POST "$VAULT_ADDR/v1/transit/encrypt/user-pii-key" \
  -H "X-Vault-Token: $VAULT_TOKEN" \
  -d '{"plaintext": "am9obi5kb2VAZXhhbXBsZS5jb20="}'
<h1>Returns: {"data":{"ciphertext":"vault:v1:8SDd3WHDOjf...","key_version":1}}</h1>

<h1>Rotate the key — old ciphertexts still decrypt, new ones use new key version</h1>
vault write -f transit/keys/user-pii-key/rotate

---

Kubernetes Auth Method

The Kubernetes auth method allows pods to authenticate using their service account JWT token. No static secrets need to be provisioned — Kubernetes identity IS the credential.

<h1>Enable Kubernetes auth</h1>
vault auth enable kubernetes

<h1>Configure with cluster details</h1>
vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  issuer="https://kubernetes.default.svc.cluster.local"

<h1>Create a role that maps K8s service accounts to Vault policies</h1>
vault write auth/kubernetes/role/api-service \
  bound_service_account_names=api-service-sa \
  bound_service_account_namespaces=production \
  policies=api-service-policy \
  ttl=1h

Vault Agent Injector Annotations

The Vault Agent Injector sidecar handles token renewal and secret delivery automatically. Add annotations to your pod spec:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "api-service"
        # Inject DB credentials as a file
        vault.hashicorp.com/agent-inject-secret-db-creds: "database/creds/api-service"
        vault.hashicorp.com/agent-inject-template-db-creds: |
          {{- with secret "database/creds/api-service" -}}
          export DB_USERNAME="{{ .Data.username }}"
          export DB_PASSWORD="{{ .Data.password }}"
          {{- end }}
        # Inject TLS certificate
        vault.hashicorp.com/agent-inject-secret-tls: "pki_int/issue/web-service"
        vault.hashicorp.com/agent-inject-template-tls: |
          {{- with secret "pki_int/issue/web-service" "common_name=api.svc.cluster.local" "ttl=24h" -}}
          {{ .Data.certificate }}
          {{- end }}
        # Auto-renew secrets before expiry
        vault.hashicorp.com/agent-pre-populate-only: "false"

---

AppRole Authentication Setup

AppRole is the recommended auth method for CI/CD systems and non-Kubernetes workloads. It issues a Role ID (public) and Secret ID (private, wrapped) that together authenticate an application.

<h1>Enable AppRole</h1>
vault auth enable approle

<h1>Create a role with tight TTLs</h1>
vault write auth/approle/role/ci-deployer \
  secret_id_ttl=10m \
  secret_id_num_uses=1 \
  token_ttl=30m \
  token_max_ttl=60m \
  token_policies=ci-deployer-policy

<h1>Get Role ID (not secret — embed in CI config)</h1>
vault read auth/approle/role/ci-deployer/role-id

<h1>Generate wrapped Secret ID (single-use, 10-minute window)</h1>
vault write -wrap-ttl=120s -f auth/approle/role/ci-deployer/secret-id

<h1>CI pipeline authenticates at runtime</h1>
VAULT_TOKEN=$(vault write -format=json auth/approle/login \
  role_id="$ROLE_ID" \
  secret_id="$WRAPPED_SECRET_ID" | jq -r .auth.client_token)

---

Secrets Manager Comparison

Feature

HashiCorp Vault

AWS Secrets Manager

Azure Key Vault

GCP Secret Manager

Dynamic credentials	Yes (DB, PKI, SSH, AWS, GCP)	No	No	No
Encryption as a service	Yes (transit engine)	No	Yes (basic)	No
Multi-cloud	Yes	AWS only	Azure only	GCP only
Self-hosted option	Yes	No	No	No
Kubernetes native auth	Yes	Via IRSA	Via Workload Identity	Via Workload Identity
Policy language	HCL (powerful)	IAM JSON	Azure RBAC	IAM JSON
Secret versioning	Yes	Yes	Yes	Yes
Automatic rotation	Yes (native)	Yes	Yes	Partial
Audit logging	Detailed (every operation)	CloudTrail	Diagnostic logs	Cloud Audit Logs
Cost	Free (OSS) / $0.03/hr (HCP)	$0.40/secret/month	$0.03/10K operations	$0.06/version/month
Compliance certifications	SOC 2 (HCP)	SOC 1/2, PCI, HIPAA	SOC 1/2, PCI, HIPAA	SOC 1/2, PCI, HIPAA

---

High Availability and Backup Strategy

<h1>Vault HA with Raft — minimum 3 nodes for fault tolerance</h1>
<h1>vault-config.hcl for each node</h1>
storage "raft" {
  path    = "/opt/vault/data"
  node_id = "vault-node-1"  # unique per node

  retry_join {
    leader_api_addr = "https://vault-node-1:8200"
  }
  retry_join {
    leader_api_addr = "https://vault-node-2:8200"
  }
  retry_join {
    leader_api_addr = "https://vault-node-3:8200"
  }
}

<h1>Take a Raft snapshot (backup)</h1>
vault operator raft snapshot save vault-backup-$(date +%Y%m%d).snap

<h1>Restore from snapshot</h1>
vault operator raft snapshot restore -force vault-backup-20260305.snap

Deploy Vault to Arxentra's managed Kubernetes platform without managing VM placement or load balancer configuration; HA still comes from your Raft configuration above, not from the platform. The Arxentra services platform handles pod scheduling, persistent volume provisioning, and health monitoring. See the deployment guide for Vault Helm chart values optimized for production.

---

Secrets Management: Getting Started Checklist

Deploy Vault with Raft storage (3 nodes minimum for HA)
Configure Auto-Unseal via cloud KMS
Enable Kubernetes auth method in every cluster
Replace all static DB passwords with dynamic credentials
Set up PKI engine to eliminate certificate management
Configure Vault Agent Injector in all namespaces
Enable audit logging to a SIEM (non-optional in production)
Schedule monthly Raft snapshot backups to encrypted object storage
Rotate root token: `vault token revoke -self` after initial setup

Every day you run with static secrets is another day a leaked git commit or compromised CI variable can grant persistent access. Dynamic credentials with 1-hour TTLs mean a leaked credential expires before an attacker can use it. Start with database dynamic credentials — that single change eliminates the most common privilege escalation path in application infrastructure.