Secret Management for Kubernetes

Kubernetes has a built-in Secret resource type. It stores key-value pairs as base64-encoded data in etcd. Base64 is not encryption -- anyone with read access to the Secret resource can decode it instantly. RBAC controls who can read Secrets, but RBAC is namespace-scoped and often overly broad. In practice, most developers in a namespace can read every Secret in that namespace.

Problems with Native Kubernetes Secrets

Base64 Is Not Encryption

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  password: czNjcjN0   # echo -n 's3cr3t' | base64

This YAML is committed to Git, stored in etcd, and readable by anyone who can kubectl get secret. The base64 encoding provides zero security -- it exists for binary safety, not confidentiality.

Coarse-Grained RBAC

Kubernetes RBAC operates at the resource level. You can grant or deny access to secrets in a namespace, but you cannot say "this service account can read db-credentials but not stripe-api-key" without creating separate namespaces for each access boundary. In practice, teams share namespaces and every pod in the namespace can read every secret.

Secret Sprawl in Manifests

Secrets referenced in pod specs, Helm charts, and Kustomize overlays proliferate across repositories. A database password might be referenced in 15 different deployment manifests. Rotating that password means updating every reference -- a manual, error-prone process.

No Audit of Secret Access

Kubernetes audit logging can capture API server reads, but most clusters do not enable Secret read auditing because of the volume of events. When they do, the logs show that a service account read a Secret, not which specific key within the Secret was used.

How KeyZero Works in Kubernetes

KeyZero runs as an init container or sidecar that resolves secrets from external backends (Vault, AWS Secrets Manager, 1Password, GCP Secret Manager) and injects them into the pod environment. Kubernetes service account tokens serve as the identity for policy evaluation.

Init Container Pattern

The init container resolves secrets and writes them to a shared volume. The application container reads them as environment variables or files.

apiVersion: v1
kind: Pod
metadata:
  name: api-server
spec:
  serviceAccountName: api-server
  initContainers:
    - name: keyzero-init
      image: getkeyzero/keyzero:latest
      command:
        - kz
        - export
        - --bundle
        - /config/bundle.yaml
        - --format
        - env
        - --output
        - /secrets/.env
      volumeMounts:
        - name: keyzero-config
          mountPath: /config
        - name: secrets
          mountPath: /secrets
      env:
        - name: KZ_JWT_PATH
          value: /var/run/secrets/kubernetes.io/serviceaccount/token

  containers:
    - name: api
      image: myapp/api-server:latest
      command: ["/bin/sh", "-c", "set -a && . /secrets/.env && exec node server.js"]
      volumeMounts:
        - name: secrets
          mountPath: /secrets
          readOnly: true

  volumes:
    - name: keyzero-config
      configMap:
        name: keyzero-bundle
    - name: secrets
      emptyDir:
        medium: Memory   # tmpfs -- secrets never hit disk

The emptyDir with medium: Memory ensures secrets are stored in RAM, not on the node's filesystem. When the pod terminates, the secrets are gone.

Sidecar Pattern for Rotation

For long-running services that need secret rotation without pod restarts, run KeyZero as a sidecar:

containers:
  - name: api
    image: myapp/api-server:latest
    envFrom:
      - secretRef:
          name: api-server-dynamic   # managed by keyzero sidecar

  - name: keyzero-sidecar
    image: getkeyzero/keyzero:latest
    command:
      - kz
      - watch
      - --bundle
      - /config/bundle.yaml
      - --interval
      - "300"
      - --output-k8s-secret
      - api-server-dynamic
    volumeMounts:
      - name: keyzero-config
        mountPath: /config

The sidecar periodically resolves secrets and updates the Kubernetes Secret resource. The application container picks up rotated values on the next read.

JWT Identity with Service Accounts

Kubernetes service accounts provide JWT tokens that KeyZero verifies as part of policy evaluation. Configure the Kubernetes issuer in your bundle:

# bundle.yaml
issuers:
  - name: k8s
    type: kubernetes
    issuer_url: "https://kubernetes.default.svc"
    audience: "keyzero"

The kubernetes issuer type maps standard K8s token claims to identity fields:

CEL Policies per Service Account

Write policies that use the service account identity for fine-grained access control:

# bundle.yaml
policies:
  - match: "secret/data/prod/db/**"
    rules:
      - cel: >
          identity.service == 'system:serviceaccount:prod:api-server'
        effect: allow
        mode: direct

  - match: "secret/data/prod/stripe/**"
    rules:
      - cel: >
          identity.service == 'system:serviceaccount:prod:payment-service'
        effect: allow
        mode: direct

  - match: "secret/data/prod/**"
    rules:
      - cel: >
          identity.env == 'prod'
          && identity.service.startsWith('system:serviceaccount:prod:')
        effect: deny   # default deny for prod

The api-server service account can access database credentials but not Stripe keys. The payment-service can access Stripe but not the database. Every other service account in the prod namespace is denied by default. This pattern is explored further in policy-based access control for machine identities.

Comparison with Native Kubernetes Secrets

Deployment

The KeyZero bundle configuration is deployed as a ConfigMap. The init container or sidecar image is published at getkeyzero/keyzero:latest. No CRDs, no operators, no cluster-level components -- KeyZero runs as a standard container within your pod spec. This means it works in managed Kubernetes environments (EKS, GKE, AKS) without requiring cluster-admin privileges or custom admission controllers. For the same approach applied to standalone containers, see the Docker integration guide. You can also pair Kubernetes with HashiCorp Vault as a backend for production secret storage. For a detailed look at how the resolution pipeline works, read the KeyZero architecture deep dive.