Kubernetes by Example · Part 6
Three database replicas with random names, fighting over one disk. Some pods need to be somebody — and some workloads aren't servers at all.
airwaves needs somewhere to keep the station list and the transcripts. You reach for what you know — a Deployment with three replicas — and it goes wrong in a way that's hard to even describe.
$ kubectl get pods -l app=catalogue
NAME READY STATUS RESTARTS
catalogue-6f9d4b7c8-k2mp4 0/1 CrashLoopBackOff 5
catalogue-6f9d4b7c8-qw8xn 1/1 Running 0
catalogue-6f9d4b7c8-z4lr9 0/1 Error 3The logs are a mess of replicas failing to find each other, two of them claiming to be primary, and at least one complaining that its data directory is locked by someone else.
Delete the broken ones and the replacements have new names, so whatever they'd agreed among themselves is void and they start arguing again.
Every one of these problems traces to the same root. A Deployment's model is that its pods are identical and disposable — and a database replica is neither.
Three things a Deployment deliberately refuses to give you, because for most workloads they'd be dead weight: a name that survives a restart, a disk that follows that name, and a predictable order of starting and stopping.
A StatefulSet gives you all three. And while we're here, the two workloads that aren't servers at all.
Deployment — pods are interchangeable. Almost everything.
DaemonSet — one per node, placed by locality. Part 5.
StatefulSet — pods have names and their own disks. Databases, queues, anything that forms a cluster.
Job — runs until it succeeds, then stops. CronJob — makes Jobs on a schedule.
Click any piece of the command.
Click any piece of the command.
Click any piece of the command.
The catalogue as a StatefulSet with its own storage, plus the nightly prune that keeps transcripts from growing forever.
The catalogue
kubectl apply -f manifests/catalogue.yaml
kubectl get sts,pods,pvc -l app=catalogue -wWatch the order. catalogue-0 appears alone and must go Ready before catalogue-1 is created at all. On a Deployment all three would have appeared at once.
One disk each
kubectl get pvc -l app=catalogue
kubectl exec catalogue-0 -- sh -c 'echo "jazz-fm" > /data/stations.txt'
kubectl exec catalogue-0 -- cat /data/stations.txtThe nightly prune, run now
kubectl apply -f manifests/prune.yaml
kubectl get cronjob prune
# do not wait until 3am
kubectl create job prune-now --from=cronjob/prune
kubectl logs job/prune-nowkubectl delete pod catalogue-0
kubectl get pods -l app=catalogue -w
# once it is Running again
kubectl exec catalogue-0 -- cat /data/stations.txtSame name, same data. Compare with Part 2, where a deleted pod was replaced by a stranger that remembered nothing. The disk did not follow the pod — it followed the ordinal, and the new pod inherited both.
kubectl scale sts catalogue --replicas=1
kubectl get pods -l app=catalogue
kubectl get pvc -l app=catalogueTwo pods gone, three PVCs still there. Kubernetes will not delete storage to satisfy a scale-down, and it is right not to.
Scale back up and the old data reappears:
kubectl scale sts catalogue --replicas=3
kubectl exec catalogue-2 -- ls -la /dataThat's either a relief or a shock. Either way, cleaning up a StatefulSet properly means deleting its PVCs by hand, and nothing will prompt you.
kubectl apply -f manifests/slow-cron.yaml # runs every minute, takes 90s
sleep 150
kubectl get jobs -l app=slow-cronTwo Jobs running at once, because concurrencyPolicy defaults to Allow and nobody chose it. For a prune job writing to shared storage, that's two processes deleting the same rows.
concurrencyPolicy: Forbid, and the lesson is that this default is wrong for most real jobs.| Because… | …this is what happens |
|---|---|
| Names are stable | Other things can depend on catalogue-0 specifically. That is what makes clustering possible. |
| Claims follow the ordinal | A replaced pod reattaches to the same data. The disk belongs to the slot, not the instance. |
| Startup is ordered | Pod 0 must be ready before 1 exists. A stuck low ordinal blocks everything above it. |
| PVCs are never auto-deleted | Scaling down and deleting leave storage behind. Clean-up is manual and unprompted. |
| Jobs are kept when finished | Logs survive for inspection, and Jobs pile up until ttlSecondsAfterFinished removes them. |
| CronJobs default to Allow | Slow runs overlap silently. Set Forbid for anything touching shared state. |
| Schedules use the cluster's clock | Set timeZone explicitly or your nightly job runs at someone else's night. |
• Use a StatefulSet when pods need identity: a stable name, their own disk, or a defined start order.
• Names are ordinals — catalogue-0, -1, -2 — and they survive restarts.
• volumeClaimTemplates gives each pod its own PVC, bound to the ordinal, not the instance.
• Creation is 0→N, deletion is N→0, and each waits for the last. podManagementPolicy: Parallel opts out.
• A headless Service (clusterIP: None) gives per-pod DNS. StatefulSets require a serviceName.
• PVCs are never deleted automatically. Scale down, delete the set — the storage stays. Clean up by hand.
• --cascade=orphan lets you recreate a StatefulSet without the pods ever going down.
• A Job runs to success, retries to backoffLimit, and is kept afterwards. Use ttlSecondsAfterFinished.
• kubectl create job X --from=cronjob/Y runs a scheduled job right now. Learn this one.
• concurrencyPolicy defaults to Allow — usually wrong. Set timeZone too.