Kubernetes by Example · Part 9
Three restarts overnight and the buffer was empty every time. The container filesystem is writable, and that is exactly the problem.
The tuner buffers a few seconds of audio to disk while the decoder catches up. Simple enough — write to /tmp/buffer, read it back.
It works for hours. Then one morning the transcripts have gaps, and the pod shows something you've learned to notice:
$ kubectl get pods -l app=tuner
NAME READY STATUS RESTARTS AGE
tuner-6b8f4d9c7-mn4kp 1/1 Running 3 (2h ago) 14hThree restarts. The pod is Running now and has been for two hours. And every restart silently threw away the buffer, because the container filesystem is not storage — it's scratch space that happens to be writable.
Nothing logged an error. From the container's point of view it wrote a file successfully and later found an empty directory, which is exactly what it was promised.
Kubernetes has a lot of volume types and only one question that matters about each: what has to die before this data is gone?
Get that right and the rest is syntax.
Container filesystem — never, for anything you want back.
emptyDir — scratch space, and sharing between containers in one pod.
hostPath — only to reach something that exists on the node: a device, a socket, shared memory.
PVC — anything that must survive the pod. Databases, uploads, state.
Click any piece of the command.
Click any piece of the command.
Click any piece of the command.
The shared memory rings that connect tuner and decoder — the piece airwaves has been describing since Part 5 and can finally implement.
Both sides, mounting the same memory
kubectl apply -f manifests/decoder-shm.yaml
kubectl apply -f manifests/tuner-shm.yaml
kubectl get pods -o wide -l 'app in (tuner,decoder)'Prove they see the same thing
# pick a tuner, note its node
POD=$(kubectl get pod -l app=tuner -o name | head -1)
NODE=$(kubectl get $POD -o jsonpath='{.spec.nodeName}')
# the decoder on that same node
DEC=$(kubectl get pod -l app=decoder --field-selector spec.nodeName=$NODE -o name)
kubectl exec $POD -- ls -la /dev/shm
kubectl exec $DEC -- ls -la /dev/shmIdentical listings, because it is literally the same memory. Now try a decoder on a different node:
kubectl get pods -o wide -l app=decoder
kubectl exec -- ls -la /dev/shm Empty. That emptiness is the entire argument for Part 5's DaemonSet, made concrete. Locality here isn't a preference — it's the difference between data existing and not.
kubectl exec deploy/tuner -- sh -c 'echo "important" > /tmp/buffer'
kubectl exec deploy/tuner -- cat /tmp/buffer
# force a restart of the container, in place
kubectl exec deploy/tuner -- kill 1
sleep 10
kubectl exec deploy/tuner -- cat /tmp/bufferNo such file or directory. Same pod, same name, same IP, restart count up by one — and the file is gone. Now the same thing on the emptyDir:
kubectl exec deploy/tuner -- sh -c 'echo "important" > /work/buffer'
kubectl exec deploy/tuner -- kill 1
sleep 10
kubectl exec deploy/tuner -- cat /work/bufferStill there. The emptyDir belongs to the pod, and the pod never went anywhere. Delete the pod, though, and it's gone — a different lifetime entirely.
kubectl apply -f manifests/tmpfs-demo.yaml
kubectl exec deploy/tmpfs-demo -- sh -c \
'dd if=/dev/zero of=/cache/big bs=1M count=200'
kubectl get pods -l app=tmpfs-demoOOMKilled, exit 137 — and the process never allocated a byte. A memory-backed emptyDir counts against the pod's memory limit, so writing files is indistinguishable from leaking memory as far as the kernel is concerned.
This one costs people entire afternoons. sizeLimit turns it into a clean disk-full error instead.
kubectl apply -f manifests/rwx-claim.yaml
kubectl get pvc rwx-demo
kubectl describe pvc rwx-demo | tail -6Pending, and the events explain it: the default StorageClass on kind does ReadWriteOnce only. Any pod using this claim sits in ContainerCreating indefinitely, and neither the pod nor the deployment mentions storage at all.
ContainerCreating with no obvious cause — go and look at its PVCs.| Because… | …this is what happens |
|---|---|
| The container filesystem dies with the container | A restart silently discards it. Never store anything you want there. |
| emptyDir belongs to the pod | Survives restarts, dies on reschedule. For scratch and for sharing between containers. |
| hostPath binds to a node | The only way to reach node-local things — and the data never follows a rescheduled pod. |
medium: Memory counts as memory | Files in a tmpfs volume can OOM-kill the pod. Set sizeLimit. |
| ReadWriteOnce means one node | Pods on the same node can share it. Pods on different nodes cannot — the second one waits. |
| A Pending PVC blocks the pod | ContainerCreating with no explanation usually means storage. Check describe pvc. |
| Reclaim policy is often Delete | Deleting a claim — or a namespace — can destroy the data with no second prompt. |
• Ask one question of every volume: what has to die before this data is gone?
• Container filesystem → dies on container restart. Not storage.
• emptyDir → dies with the pod. For scratch, and for sharing between containers in one pod.
• hostPath → the node's own disk. Only for node-local things; the data never travels.
• PVC → outlives the pod, the workload, and usually your intentions.
• medium: Memory is tmpfs and counts against the memory limit — files can cause an OOM kill.
• ReadWriteOnce means one node, not one pod. RWX needs a network filesystem you may not have.
• Every container mounts separately, init containers included. Declaring a volume mounts nothing.
• Pending PVC → ContainerCreating pod, with no clue at the pod level. describe pvc.
• hostPath + non-root containers = permission problems. That's what fix-perms init containers are for.