Nidhi's blog

Production Engineering • SRE • Kubernetes

Key and lock security illustration

Syncing Kubernetes Secrets to Google Secret Manager

July 8, 2026

The short version

Kubernetes secrets are base64-encoded, not encrypted, and tied to your cluster. We wanted them in Google Secret Manager for IAM-gated access, audit logs, and a fallback that survives cluster loss.

  1. A bulk script copies all secrets across clusters into GSM using a consistent naming convention, skipping system noise (SA tokens, Helm releases, bootstrap tokens)
  2. A daily CronJob keeps GSM current using Workload Identity - no static credentials
  3. Apps migrate to reading from GSM gradually while the CronJob bridges the gap

Why not just leave secrets in Kubernetes

By default, secrets are stored as base64 in etcd - not encrypted at rest unless you’ve explicitly configured envelope encryption with a KMS provider. Anyone with etcd access or a broad get secrets ClusterRole can read everything. Access control is coarse: RBAC can limit which pods can read which secrets, but there’s no per-access audit trail.

GSM gives you a different set of properties: every access is logged in Cloud Audit Logs, IAM controls are per-secret, you get versioning and automatic rotation hooks, and anything with Workload Identity can reach it without a kubeconfig. Critically, GSM lives outside the cluster. If you need to rebuild from scratch, your secrets are still there.

We didn’t want to remove secrets out of Kubernetes immediately, in case of failures, it turns into production incident. Instead we built a sync layer: keep writing to k8s, mirror to GSM continuously, migrate apps to read from GSM one by one.

The naming convention

GSM secret names have constraints: lowercase letters, digits, hyphens, and underscores only, max 255 characters. We encode the full origin context into the name:

k8s--<cluster>--<namespace>--<secret-name>

For example, a secret named postgres-credentials in the production namespace of cluster prod-eu becomes:

k8s--prod-eu--production--postgres-credentials

The script normalises everything - uppercase to lowercase, disallowed characters to hyphens, truncated at 255:

gsm_name() {
    echo "k8s--${1}--${2}--${3}" \
        | tr '[:upper:]' '[:lower:]' \
        | sed 's/[^a-z0-9_-]/-/g' \
        | cut -c1-255
}

The double-hyphen separator is deliberate. Namespace and secret names can contain single hyphens, so a single separator would be ambiguous. Double-hyphen can’t appear naturally in either field.

The bulk migration script

The script iterates every context, every namespace, every secret, and mirrors it to GSM. The core logic is straightforward: check if the GSM secret exists, then either create or add a new version.

json_payload=$(kubectl get secret "$secret_name" -n "$ns" -o json | python3 -c "
import sys, json, base64
s = json.load(sys.stdin)
data = s.get('data') or {}
out = {}
for k, v in data.items():
    try:
        out[k] = base64.b64decode(v).decode('utf-8')
    except Exception:
        out[k] = v
print(json.dumps(out, indent=2))
")

if gcloud secrets describe "$gsm_secret" --project="$PROJECT" &>/dev/null; then
    echo -n "$json_payload" | gcloud secrets versions add "$gsm_secret" \
        --project="$PROJECT" --data-file=- --quiet
else
    echo -n "$json_payload" | gcloud secrets create "$gsm_secret" \
        --project="$PROJECT" --replication-policy="automatic" \
        --labels="source=k8s,cluster=${cluster_label},namespace=${ns}" \
        --data-file=- --quiet
fi

Each secret is stored as a JSON object of decoded key-value pairs. The .data field in a Kubernetes secret is a map of base64-encoded values - we decode each one. Binary values that can’t be decoded to UTF-8 are kept as-is.

What gets skipped. Three types are noise and skipped automatically:

  • kubernetes.io/service-account-token - auto-rotated, useless outside the cluster
  • helm.sh/release.v1 - gzipped+base64 Helm release blobs, not useful in GSM
  • bootstrap.kubernetes.io/token - short-lived cluster bootstrap tokens
  • Names matching default-token-* or sh.helm.release.*

Empty secrets (no .data keys) are also skipped.

Run with --dry-run first to preview every secret that would be created:

./k8s-secrets-to-gsm.sh \
  --project my-gcp-project \
  --contexts prod-eu,staging-eu \
  --dry-run

Remove --dry-run to write for real. Add --namespaces production,monitoring to scope to specific namespaces.

Keeping them in sync: the CronJob

The bulk script is a one-shot. To keep GSM current as secrets change in the cluster, we run a CronJob daily at 2am:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: secret-gsm-syncer
  namespace: kube-system
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: secret-gsm-syncer
          restartPolicy: OnFailure
          containers:
            - name: syncer
              image: google/cloud-sdk:slim

concurrencyPolicy: Forbid means if a run is still going when the next one is due (unlikely, but possible on a large cluster), the new run is skipped rather than running in parallel.

Workload Identity instead of static credentials. The pod’s Kubernetes ServiceAccount is annotated to map to a GCP service account:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: secret-gsm-syncer
  namespace: kube-system
  annotations:
    iam.gke.io/gcp-service-account: secret-gsm-syncer@<your-project>.iam.gserviceaccount.com

No JSON key file, no secret containing a secret. The GCP SA needs roles/secretmanager.admin - or more narrowly, secretmanager.secrets.create and secretmanager.versions.add if you want to avoid granting delete.

The ClusterRole is read-only:

rules:
  - apiGroups: [""]
    resources: ["secrets", "namespaces"]
    verbs: ["get", "list"]

The sync logic inside the container is the same create/update pattern as the bulk script, just embedded in the pod’s command.

The migration path

Don’t try to do this atomically. The gradual path is safer:

  1. Dry-run first. Understand the volume - how many secrets, which namespaces, any naming collisions. Fix anything surprising before writing.

  2. Run the bulk migration. This populates GSM with everything currently in the cluster. Label source=k8s is set on every secret for easy filtering.

  3. Enable the CronJob. From this point GSM stays current automatically. Apps can still read from Kubernetes - nothing has changed for them yet.

  4. Migrate apps one by one. Update each deployment to read from GSM using the SDK or mounted via the Secret Manager sidecar. The CronJob ensures GSM has the latest value while you’re mid-migration.

  5. Clean up. Once all apps read from GSM, the Kubernetes secrets are redundant. You can leave them as a fallback or clean up - the CronJob stops writing new versions to GSM if you delete the source secret (it simply won’t find it).

When it breaks

CronJob pods failing with permission errors.

Check whether Workload Identity is correctly configured. The GCP SA binding needs to exist in both directions: the iam.gke.io/gcp-service-account annotation on the k8s SA, and an IAM binding granting the k8s SA the roles/iam.workloadIdentityUser role on the GCP SA. Missing either side means the pod can’t authenticate.

Secret name collision in GSM.

Two secrets in different namespaces or clusters that produce the same normalised name. The -- separator makes this unlikely but not impossible if names are very long and get truncated. Check the dry-run output for duplicate GSM names before running for real.

Binary secret values not syncing correctly.

If a secret contains a binary value (a TLS key, a keystore), the Python decode step keeps it as base64. When your app reads it from GSM it’ll get the base64 string, not the raw bytes. Handle this at the read side or pre-process the value before storing.

GSM is current but the app still reads stale values.

Apps that cache the secret at startup won’t see updates until they restart. If you need live rotation, use the Secret Manager sidecar or read from GSM on every request with a short TTL cache.

CronJob ran but some secrets weren’t updated.

Check for secrets the skip logic matched unexpectedly - grep the job logs for EMPTY or skipped. Also verify the ClusterRole binding is cluster-scoped, not namespace-scoped; a namespaced RoleBinding won’t grant access to secrets in other namespaces.