Secrets

Secrets Command

Structured guidance for secrets strategy, External Secrets Operator, Sealed Secrets, rotation, and Kubernetes-side secrets audit.

Activation

Copy code
/platform-skills:secrets design   # ESO vs Sealed Secrets decision; choose a strategy for the cluster
/platform-skills:secrets eso      # ExternalSecret/SecretStore scaffold; debug sync errors
/platform-skills:secrets sealed   # seal, rotate, backup master key; troubleshoot decryption failures
/platform-skills:secrets rotate   # rotation runbook — update provider value, force sync, verify pods reload
/platform-skills:secrets audit    # find bad SA tokens, unhealthy ExternalSecrets, missing secrets

Interactive Wizard (fires when no mode is provided)

When invoked with no arguments, ask before proceeding:

Q1 — Mode?

Copy code
What do you need?
  1. design   — choose ESO vs Sealed Secrets for a cluster or workload
  2. eso      — scaffold ExternalSecret/SecretStore, or debug a sync error
  3. sealed   — seal a new secret, rotate an existing one, or back up the master key
  4. rotate   — end-to-end rotation runbook for a live secret
  5. audit    — find service account token secrets, unhealthy ExternalSecrets, plain secrets in Git

Enter 1–5 or mode name:

Q2 — Context (after mode selected):

• design: What is the backend — AWS Secrets Manager, Azure Key Vault, HashiCorp Vault, or no cloud provider?
• eso: Paste the ExternalSecret or SecretStore YAML, or describe the sync error.
• sealed: New secret or rotating an existing one? Which namespace?
• rotate: Which secret is rotating — database password, API key, TLS cert? Which backend?
• audit: Cluster name and namespace scope. Any specific concern — Git scan, SA tokens, ESO health?

Mode: design

Triggers: which pattern, ESO vs Sealed Secrets, choose secrets strategy, what should I use

Read references/secrets.md before responding.

Decision matrix

Pick ESO when:

• Cluster has workload identity (IRSA on EKS, Azure Workload Identity on AKS, Vault Kubernetes auth)
• Rotation should propagate automatically without a Git commit
• Secrets are managed centrally in a provider (team already uses AWS SM / Azure KV / Vault)

Pick Sealed Secrets when:

• No cloud provider or Vault
• Air-gapped environment
• Team prefers all cluster state — including encrypted secrets — reviewable in Git without a runtime backend

Handoffs:

• For filesystem or Git history scanning for leaked secrets → /platform-skills:trivy (--scanners secret)
• For Azure Key Vault identity setup → /platform-skills:azure identity
• For AWS IRSA setup → /platform-skills:aws

Mode: eso

Triggers: ExternalSecret, SecretStore, ClusterSecretStore, sync error, ESO, external secrets operator, SecretSyncError

Read references/secrets.md → External Secrets Operator section before responding.

Before generating commands, confirm:

• ESO namespace — common values: external-secrets, platform-system, kube-system. Discover with: kubectl get pods -A | grep external-secrets
• ESO deploy name — common values: external-secrets, external-secrets-controller. Discover with: kubectl get deploy -n <eso-namespace>
• ESO service account name — the SA annotated with IRSA or Workload Identity. Common: external-secrets-sa, but teams name it differently.

Substitute these into all namespace flags, kubectl logs, and serviceAccountRef fields below.

ExternalSecret scaffold (provider-agnostic)

yamlCopy code
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: app-team
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: my-secret-store
    kind: SecretStore
  target:
    name: database-credentials
    creationPolicy: Owner
  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: prod/app-team/db
        property: password

SecretStore — provider examples

AWS Secrets Manager (IRSA):

yamlCopy code
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: my-secret-store
  namespace: app-team
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-north-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa

Required IAM (least privilege):

jsonCopy code
{
  "Effect": "Allow",
  "Action": "secretsmanager:GetSecretValue",
  "Resource": "arn:aws:secretsmanager:<region>:<account>:secret:prod/app-team/*"
}

Azure Key Vault (Workload Identity):

yamlCopy code
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: my-secret-store
  namespace: app-team
spec:
  provider:
    azurekv:
      authType: WorkloadIdentity
      vaultUrl: "https://my-keyvault.vault.azure.net"
      serviceAccountRef:
        name: external-secrets-sa

The managed identity needs Key Vault Secrets User on the Key Vault.

HashiCorp Vault (Kubernetes auth):

yamlCopy code
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: my-secret-store
  namespace: app-team
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "app-team-role"
          serviceAccountRef:
            name: external-secrets-sa

Debug ESO sync errors

bashCopy code
# Sync status and last sync time
kubectl get externalsecret -n app-team database-credentials

# Full status including error message
kubectl describe externalsecret -n app-team database-credentials

# Operator logs for this secret
kubectl logs -n external-secrets deploy/external-secrets \
  | grep -i "database-credentials"

# SecretStore health
kubectl get secretstore -n app-team my-secret-store
kubectl describe secretstore -n app-team my-secret-store

Mode: sealed

Triggers: SealedSecret, kubeseal, seal, sealed secrets, no key could decrypt, master key, backup

Read references/secrets.md → Sealed Secrets section before responding.

Interview — ask before generating any commands

Before writing any kubeseal or kubectl commands, collect the following. Different organisations deploy the controller with different names and namespaces.

Copy code
Q1 — Controller namespace:
  Where is the sealed-secrets controller running?
  (common: sealed-secrets, kube-system, platform-system — check with: kubectl get pods -A | grep sealed)

Q2 — Controller Service name:
  What is the controller Service name? (used by kubeseal --controller-name — this is the Service, not the Deployment)
  (common: sealed-secrets, sealed-secrets-controller — check with: kubectl get svc -n <namespace> | grep sealed)

Q3 — Secret namespace:
  Which namespace will the unsealed Secret live in?

Q4 — Secret name:
  What is the name of the secret?

Q5 — Action:
  1. Seal a new secret
  2. Rotate an existing sealed secret
  3. Back up the master key
  4. Restore the master key
  5. Troubleshoot a decryption failure

Use the answers to substitute <controller-namespace>, <controller-name>, and <secret-namespace> in all commands below. Never output hardcoded namespace or deployment names — always use the values the user provided.

Note: <controller-name> means the Service name when used with kubeseal --controller-name, and the Deployment name when used with kubectl rollout restart deploy/<controller-name>. If they differ in your cluster, substitute accordingly.

Seal a new secret

bashCopy code
# Fetch the cluster public key — run once per cluster or after key rotation
kubeseal --fetch-cert \
  --controller-name=<controller-name> \
  --controller-namespace=<controller-namespace> \
  > pub-cert.pem

# Create a plain Secret and immediately seal it — never commit the plain Secret
kubectl create secret generic <secret-name> \
  --namespace=<secret-namespace> \
  --from-literal=KEY=value \
  --dry-run=client -o yaml \
  | kubeseal --cert pub-cert.pem --format yaml \
  > <secret-name>-sealed.yaml

git add <secret-name>-sealed.yaml
git commit -m "chore: seal <secret-name> for <secret-namespace>"

Rotate an existing sealed secret

bashCopy code
kubectl create secret generic <secret-name> \
  --namespace=<secret-namespace> \
  --from-literal=KEY=newvalue \
  --dry-run=client -o yaml \
  | kubeseal --cert pub-cert.pem --format yaml \
  > <secret-name>-sealed.yaml

git add <secret-name>-sealed.yaml
git commit -m "chore: rotate <secret-name>"

After GitOps applies the updated SealedSecret:

• Pods using a mounted volume pick up the new value automatically via kubelet refresh
• Pods using env vars require a rollout restart: kubectl rollout restart deployment/<name> -n <secret-namespace>

Back up the master key

bashCopy code
kubectl get secret -n <controller-namespace> \
  -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o yaml > sealed-secrets-master-key-backup.yaml

Store this file outside the cluster (password manager, vault, encrypted object store). Without it, sealed secrets cannot be decrypted after cluster recreation.

Restore the master key

bashCopy code
kubectl apply -f sealed-secrets-master-key-backup.yaml
kubectl rollout restart deploy/<controller-name> -n <controller-namespace>

Troubleshoot

bashCopy code
# Confirm the controller is running
kubectl get pods -n <controller-namespace>

# Inspect SealedSecret status
kubectl describe sealedsecret <secret-name> -n <secret-namespace>

# Controller logs for decryption errors
kubectl logs -n <controller-namespace> deploy/<controller-name> | tail -50

Mode: rotate

Triggers: rotate, rotation runbook, change password, new token, expired credential, update secret

Apply this runbook for any secret rotation:

Step 1 — Update the value in the provider

AWS Secrets Manager:

bashCopy code
aws secretsmanager put-secret-value \
  --secret-id prod/app-team/db \
  --secret-string '{"password":"new-password"}' \
  --profile <aws-profile>

Azure Key Vault:

bashCopy code
az keyvault secret set \
  --vault-name my-keyvault \
  --name db-password \
  --value "new-password"

Sealed Secrets: Re-seal with the new value (see sealed mode).

Step 2 — Force sync (ESO)

bashCopy code
kubectl annotate externalsecret database-credentials \
  force-sync=$(date +%s) \
  -n app-team

Step 3 — Verify the Kubernetes Secret was updated

bashCopy code
# Check the last sync time
kubectl get externalsecret database-credentials -n app-team

# Confirm the Secret data changed (check hash, not the value)
kubectl get secret database-credentials -n app-team \
  -o jsonpath='{.data.DB_PASSWORD}' | base64 -d | sha256sum

Step 4 — Reload pods if needed

bashCopy code
# Pods using env vars require a restart to pick up the new Secret value
kubectl rollout restart deployment/<name> -n app-team
kubectl rollout status deployment/<name> -n app-team

Pods using mounted volumes reload automatically within the kubelet sync period (default 60s) — no restart needed unless the app reads the value at startup only.

Step 5 — Validate the application is using the new credential

bashCopy code
# Application-specific validation — confirm connectivity to the backend
# Do NOT print secret values; test the connection functionally instead
kubectl exec -n app-team deploy/<name> -- \
  sh -c 'wget -qO- http://localhost:8080/healthz || curl -sf http://localhost:8080/healthz'
# For a database: test the connection, not the credential value
kubectl exec -n app-team deploy/<name> -- \
  sh -c 'pg_isready -h $DB_HOST -p $DB_PORT' 2>/dev/null || echo "adjust command for your backend"

Mode: audit

Triggers: audit, find plain secrets, SA token, kubernetes.io/service-account-token, secrets health, hygiene

This mode covers Kubernetes-side audit only. For filesystem and Git history scanning for leaked secrets → /platform-skills:trivy (fast, offline pattern check) or /platform-skills:kingfisher (adds live validation against the provider, blast-radius mapping, and revocation — reach for this when the question is "is it still live" rather than "does it match a pattern").

Find workloads using legacy SA token secrets

bashCopy code
# Find all kubernetes.io/service-account-token type secrets
kubectl get secrets -A \
  --field-selector type=kubernetes.io/service-account-token \
  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,SA:.metadata.annotations.kubernetes\.io/service-account\.name'

Legacy manually-created SA token secrets do not auto-rotate. Migrate workloads to projected tokens (automatic since Kubernetes 1.24).

Check ExternalSecret sync health across namespaces

bashCopy code
# All ExternalSecrets and their status
kubectl get externalsecrets -A \
  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,STATUS:.status.conditions[0].reason,LAST:.status.refreshTime'

# Find any not in Ready state
kubectl get externalsecrets -A \
  -o json \
  | jq -r '.items[] | select(.status.conditions[]?.type=="Ready" and .status.conditions[]?.status!="True") | "\(.metadata.namespace)/\(.metadata.name): \(.status.conditions[]?.message)"'

Check for plain secrets committed to Git

bashCopy code
# Trivy handles a fast, offline first pass — hand off explicitly
# trivy repo --scanners secret <repo-url>
# or for a local clone:
# trivy fs --scanners secret .

# Kingfisher adds live validation, blast-radius mapping, and revocation
# kingfisher scan <repo-url-or-path>

For a Git history and filesystem scan → /platform-skills:trivy. For live validation of what's found, or revocation of a confirmed-live credential → /platform-skills:kingfisher

Check SecretStore health

bashCopy code
kubectl get secretstores -A
kubectl get clustersecretstores

# Describe any not-ready stores
kubectl get secretstores -A -o json \
  | jq -r '.items[] | select(.status.conditions[]?.status!="True") | "\(.metadata.namespace)/\(.metadata.name): \(.status.conditions[]?.message)"'

Operational rules to validate

• No plain Secret manifests committed to Git — verify with git log --all -- '*.yaml' | xargs grep -l 'kind: Secret'
• All workloads use dedicated service accounts, not default
• No kubernetes.io/service-account-token type secrets for application API auth
• ClusterSecretStore used only for platform-wide secrets; namespace-scoped SecretStore for everything else
• Provider permissions scoped to minimum required paths — not Resource: "*" or all secrets in the vault

Common mistakes

• Committing plain Secret manifests to Git — use ESO, Sealed Secrets, or Vault. Rotate any secret that was ever in Git history.
• secretsmanager:GetSecretValue with Resource: "*" — scope to the specific path or secret name. Wildcard grants access to all secrets in the account.
• Using default service account for ESO — creates implicit access. Create a dedicated external-secrets-sa per namespace.
• Sealed Secrets rotation without --fetch-cert — if the controller key rotated since the last seal, the new SealedSecret cannot be decrypted. Always fetch the current cert before sealing.
• Skipping the master key backup before cluster destruction — without the backup, all sealed secrets in Git are permanently unreadable on the new cluster.
• Relying on env var pod restart for mounted volume secrets — mounted volume secrets refresh automatically; env vars do not. Know which pattern each workload uses before planning rotation.

Reference

Full guidance: references/secrets.md

For filesystem and Git history secret scanning: /platform-skills:trivy

For Azure Key Vault identity setup: /platform-skills:azure identity

For AWS IRSA setup: /platform-skills:aws