Seeking Wisdom · AI/ML Engineering · Investing · Books · Projects · About ·

Kubernetes by Example · Part 8

Config and Secrets

You edited the ConfigMap. You confirmed it changed. The pod carried on with the old value, and will do so forever.

assumes Parts 1–7~17 minutesthe most common surprise

The symptom

The station URLs live in a ConfigMap now, which feels like progress. A URL changes, so you edit the ConfigMap:

$ kubectl edit configmap stations
configmap/stations edited

$ kubectl get configmap stations -o yaml | grep jazz
  jazz.url: http://new-url.example/stream

Edited. Confirmed. And the tuner carries on using the old URL.

You wait a minute, in case something is eventually consistent. Then five. You check the ConfigMap again — still the new value. You exec into the pod:

$ kubectl exec deploy/tuner -- env | grep JAZZ
JAZZ_URL=http://old-url.example/stream

The old value, inside the container, with the new value sitting in the ConfigMap three feet away.

Nothing is broken. This is exactly what you asked for — you just didn't know you were asking for it, because the two ways of getting config into a pod look almost identical in YAML and behave nothing alike.

Two doors, very different rooms

A ConfigMap is a bag of strings; a Secret is the same bag with slightly different handling. Neither is interesting on its own. What matters is how a pod consumes them, because that choice decides whether your config can ever change without a restart.

ConfigMap · stations jazz.url: http://… news.url: http://… plain text, 1 MiB limit, no secrecy Secret · station-token token: c2VjcmV0LXZhbHVl base64 — an encoding, not a lock anyone who can read it, can read it Pod · tuner as environment variables JAZZ_URL=http://… copied once, when the container starts edit the ConfigMap afterwards and nothing happens. Ever. as a mounted volume /etc/stations/jazz.url the kubelet refreshes it, within about a minute your process still has to notice the file changed downward API — the pod reads itself POD_NAME, NODE_NAME, the pod's own labels one-time copy kept in sync frozen at startup — only a restart changes it …unless you used subPath, which silently freezes it too

Choosing between them

Environment variables — frozen at startup, visible in kubectl describe, inherited by child processes. Right when the value never changes without a restart anyway.

Mounted volume — refreshed by the kubelet, and your process has to re-read. Right for large files, certificates, and anything you want to reload live.

Either way, the safe default is to restart on change. A checksum annotation makes that automatic.

The commands

Making ConfigMaps without writing YAML

Click any piece of the command.

Secrets, and reading them back

Click any piece of the command.

Wiring it into a pod

Click any piece of the command.

Build it

The station list as a ConfigMap, a token as a Secret, and the tuner learning its own name.

Config in, hardcoding out

kubectl apply -f manifests/stations-cm.yaml
kubectl apply -f manifests/tuner.yaml
kubectl rollout status deploy/tuner

Both paths, side by side

# env vars — the frozen copy
kubectl exec deploy/tuner -- env | grep -E 'STATION|POD_|NODE_'

# the same ConfigMap, mounted as files
kubectl exec deploy/tuner -- ls -la /etc/stations/
kubectl exec deploy/tuner -- cat /etc/stations/jazz.url

POD_NAME and NODE_NAME came from the downward API — nobody passed them in. The pod asked the cluster about itself.

And the Secret, decoded in one line

kubectl get secret station-token -o jsonpath='{.data.token}' | base64 -d; echo

That is the whole of it. If this command alarms you, good — the protection was never the encoding, it's who is allowed to run it. Part 13.

Break it

1 · Reproduce the symptom, then watch the other path work

kubectl patch configmap stations --type=merge \
  -p='{"data":{"jazz.url":"http://changed.example/stream"}}'

# the env var: unchanged, now and forever
kubectl exec deploy/tuner -- env | grep JAZZ

# the mounted file: give it a minute
sleep 70
kubectl exec deploy/tuner -- cat /etc/stations/jazz.url

Same ConfigMap, same pod, same moment. The environment variable is a photograph taken at startup. The file is a live mirror. Nothing in the YAML signals this difference, which is why it catches people.

2 · Discover that subPath doesn't refresh

kubectl apply -f manifests/subpath-demo.yaml
kubectl patch configmap stations --type=merge \
  -p='{"data":{"jazz.url":"http://again.example/stream"}}'
sleep 70
kubectl exec deploy/subpath-demo -- cat /etc/jazz.url

Still the old value. subPath looks like a tidier volume mount and quietly gives you env-var semantics. If you've ever mounted a single config file and wondered why it never updates, this is why.

3 · Remove a key a pod depends on

kubectl patch configmap stations --type=json \
  -p='[{"op":"remove","path":"/data/jazz.url"}]'
kubectl rollout restart deploy/tuner
kubectl get pods -l app=tuner

CreateContainerConfigError. The pod will not start at all, and kubectl describe pod names the missing key exactly. This is the right failure — loud, immediate, specific — rather than starting with an empty string and failing somewhere confusing later.

optional: true turns this off, which you should only do when the pod genuinely works without the value.

What follows from this

Because……this is what happens
Env vars resolve onceChanging a ConfigMap never reaches a running container. Restart, or use a volume.
Volumes are refreshedFiles update within about a minute — but your process must re-read them.
subPath bypasses the refreshA single-file mount is silently frozen. Mount the directory instead.
Secrets are base64One pipe decodes them. The real protection is RBAC, which is Part 13.
ConfigMaps live in etcd1 MiB ceiling. Not a place for models, bundles or archives.
A missing key blocks startupCreateContainerConfigError — loud and specific, which is what you want.
Template changes trigger rolloutsA checksum annotation turns a config edit into an automatic, paced restart.

Cheat sheet

Part 8 · config and secrets

• Env vars are frozen at container start. Editing the ConfigMap afterwards changes nothing, ever.

• Mounted volumes refresh within ~a minute — but your process still has to re-read the file.

• subPath silently disables refresh. Mount the directory if you want updates.

• A Secret is base64, not encryption. … -o jsonpath='{.data.x}' | base64 -d is the whole story.

• The real protection for Secrets is RBAC — who can get them. Check with kubectl auth can-i.

• ConfigMaps cap at 1 MiB. They live in etcd.

• A missing referenced key gives CreateContainerConfigError. Use optional: true only when it's truly optional.

• The downward API lets a pod read its own name, node, labels and limits — fieldRef, resourceFieldRef.

• kubectl create … --dry-run=client -o yaml is how you write these without hand-editing base64.

• To make a change land: rollout restart, or a checksum annotation so Part 4 does it for you.