Managing Secrets

Managing Secrets

Secure storage, rotation, and delivery of secrets (API keys, database credentials, TLS certificates) for applications and infrastructure.

When to Use This Skill

Use when:

• Storing API keys, database credentials, or encryption keys
• Implementing secret rotation (manual or automatic)
• Syncing secrets from external stores to Kubernetes
• Setting up dynamic secrets (database, cloud providers)
• Scanning code for leaked secrets
• Implementing zero-knowledge patterns
• Meeting compliance requirements (SOC 2, ISO 27001, PCI DSS)

Quick Decision Frameworks

Framework 1: Choosing a Secret Store

Decision Tree:

• Kubernetes? → External Secrets Operator (ESO) with chosen backend
• Single cloud? → Cloud-native (AWS/GCP/Azure)
• Multi-cloud/on-prem? → HashiCorp Vault
• GitOps? → SOPS or Sealed Secrets

Framework 2: Static vs. Dynamic Secrets

Framework 3: Kubernetes Secret Delivery

HashiCorp Vault Fundamentals

Core Components

• Secrets Engines: KV v2 (static), Database (dynamic), AWS, PKI, SSH
• Auth Methods: Kubernetes, JWT/OIDC, AppRole, LDAP
• Policies: HCL-based access control (least privilege)
• Leases: TTL for secrets, auto-renewal, auto-revocation

Static Secrets (KV v2)

bashCopy code
# Create secret
vault kv put secret/myapp/config api_key=sk_live_EXAMPLE

# Read secret
vault kv get secret/myapp/config

# List versions
vault kv metadata get secret/myapp/config

Dynamic Database Credentials

bashCopy code
# Configure PostgreSQL
vault write database/config/postgres \
  plugin_name=postgresql-database-plugin \
  connection_url="postgresql://{{username}}:{{password}}@postgres:5432/mydb"

# Create role
vault write database/roles/app-role \
  db_name=postgres \
  creation_statements="CREATE ROLE \"{{name}}\"..." \
  default_ttl="1h"

# Generate credentials
vault read database/creds/app-role

For detailed Vault architecture, see references/vault-architecture.md.

Kubernetes Integration

External Secrets Operator (ESO)

Syncs secrets from 30+ providers to Kubernetes Secrets.

yamlCopy code
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      auth:
        kubernetes:
          role: "app-role"

yamlCopy code
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
  target:
    name: db-credentials
  data:
  - secretKey: password
    remoteRef:
      key: secret/data/database/config

Vault Secrets Operator (VSO)

Kubernetes-native Vault integration with automatic lease renewal.

yamlCopy code
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  name: postgres-creds
spec:
  vaultAuthRef: vault-auth
  mount: database
  path: creds/app-role
  renewalPercent: 67  # Renew at 67% of TTL
  destination:
    name: dynamic-db-creds

For ESO vs CSI vs VSO comparison, see references/kubernetes-integration.md.

Secret Rotation Patterns

Pattern 1: Versioned Static Secrets (Blue/Green)

1. Create new secret version in Vault
2. Update staging environment
3. Monitor for errors (24-48 hours)
4. Gradual production rollout (10% → 50% → 100%)
5. Revoke old secret (after 7 days)

Pattern 2: Dynamic Database Credentials

Vault auto-generates credentials with short TTL:

• App fetches credentials from Vault
• Vault automatically renews lease (at 67% of TTL)
• On expiration, Vault revokes access
• On renewal failure, app requests new credentials

Pattern 3: TLS Certificate Rotation

Using cert-manager + Vault PKI:

• cert-manager requests certificate from Vault
• Automatically renews before expiration (default: 67% of duration)
• Updates Kubernetes Secret on renewal
• Optional pod restart (via Reloader)

For detailed rotation workflows, see references/rotation-patterns.md.

Multi-Language Integration

Python (hvac)

pythonCopy code
import hvac

client = hvac.Client(url='https://vault.example.com')
client.auth.kubernetes(role='app-role', jwt=jwt)

# Fetch dynamic credentials
response = client.secrets.database.generate_credentials(name='postgres-role')
username = response['data']['username']
password = response['data']['password']

Go (Vault API)

goCopy code
import vault "github.com/hashicorp/vault/api"

client, _ := vault.NewClient(vault.DefaultConfig())
k8sAuth, _ := auth.NewKubernetesAuth("app-role")
client.Auth().Login(context.Background(), k8sAuth)

secret, _ := client.Logical().Read("database/creds/postgres-role")

TypeScript (node-vault)

typescriptCopy code
import vault from 'node-vault';

const client = vault({ endpoint: 'https://vault.example.com' });
await client.kubernetesLogin({ role: 'app-role', jwt });

const response = await client.read('database/creds/postgres-role');

For complete examples, see examples/dynamic-db-credentials/.

Secret Scanning

Pre-Commit Hooks (Gitleaks)

bashCopy code
# Install Gitleaks
brew install gitleaks

# Run on staged files
gitleaks protect --staged --verbose

Pre-commit hook prevents secrets from being committed. For setup, see examples/secret-scanning/pre-commit.

CI/CD Integration

yamlCopy code
# GitHub Actions
- name: Run Gitleaks
  uses: gitleaks/gitleaks-action@v2

Remediation Workflow

When a secret is leaked:

1. Rotate immediately (within 1 hour)
2. Revoke at provider
3. Remove from Git history (BFG Repo-Cleaner)
4. Force push (notify team)
5. Audit access (who had access during leak window)
6. Document incident

For detailed remediation, see references/secret-scanning.md.

Zero-Knowledge Patterns

Client-Side Encryption (E2EE)

User password → PBKDF2 → encryption key → encrypt secret → send to server

Server stores only encrypted blobs (cannot decrypt).

Shamir's Secret Sharing

Split secret into N shares, require M to reconstruct (e.g., 3 of 5).

bashCopy code
# Initialize Vault with Shamir shares
vault operator init -key-shares=5 -key-threshold=3

# Unseal requires 3 of 5 key shares
vault operator unseal <KEY_1>
vault operator unseal <KEY_2>
vault operator unseal <KEY_3>

For implementations, see references/zero-knowledge.md.

Library Recommendations (2025)

Secret Stores

Secret Scanning

Client Libraries

Common Workflows

Workflow 1: Vault + ESO on Kubernetes

1. Install Vault (Helm chart)
2. Initialize and unseal Vault
3. Enable Kubernetes auth
4. Install External Secrets Operator
5. Create SecretStore (Vault connection)
6. Create ExternalSecret (secret mapping)

For step-by-step guide, see examples/vault-eso-setup/.

Workflow 2: Dynamic Database Credentials

1. Enable database secrets engine
2. Configure database connection
3. Create role with TTL
4. App fetches credentials
5. Vault auto-renews lease

For implementation, see examples/dynamic-db-credentials/.

Workflow 3: Secret Scanning Remediation

1. Gitleaks detects secret
2. Block commit (pre-commit hook)
3. Developer removes secret
4. Developer stores in Vault
5. Developer references Vault path

For setup, see examples/secret-scanning/.

Integration with Related Skills

• auth-security: OAuth client secrets, JWT signing keys
• databases-*: Dynamic database credentials
• deploying-applications: Container registry credentials
• observability: Grafana/Datadog API keys
• infrastructure-as-code: Cloud provider credentials

Security Best Practices

1. Never commit secrets to Git (use Gitleaks pre-commit hook)
2. Use dynamic secrets where possible
3. Rotate secrets regularly (quarterly for static, hourly for dynamic)
4. Implement least privilege (Vault policies, RBAC)
5. Enable audit logging
6. Encrypt at rest (Vault storage, etcd encryption)
7. Use short TTLs (< 24 hours for dynamic secrets)
8. Monitor failed access attempts

Common Pitfalls

Secrets in Environment Variables

Environment variables visible in process lists. Solution: Use file-based secrets (Kubernetes volumes, CSI driver).

Hardcoded Secrets in Manifests

Base64 is not encryption. Solution: Use External Secrets Operator.

No Secret Rotation

Stale credentials increase breach risk. Solution: Use dynamic secrets or automate rotation.

Root Token in Production

Unlimited permissions. Solution: Use auth methods with least privilege policies.

For Detailed Information, See

• references/vault-architecture.md - Vault internals, HA setup, policies
• references/kubernetes-integration.md - ESO, CSI driver, VSO comparison
• references/rotation-patterns.md - Detailed rotation workflows
• references/secret-scanning.md - Gitleaks, remediation procedures
• references/zero-knowledge.md - E2EE, Shamir's secret sharing
• references/cloud-providers.md - AWS, GCP, Azure secret managers
• examples/vault-eso-setup/ - Complete Kubernetes setup
• examples/dynamic-db-credentials/ - Multi-language examples
• examples/secret-scanning/ - Pre-commit hooks, CI/CD
• scripts/setup_vault.sh - Automated Vault installation