Kubernetes by Example · Part 12
You added a health check and it killed every pod you had. Only one of the three probes can do that, and it was the one you picked.
You add a liveness probe to the tuner, because adding health checks is obviously good practice.
It's fine for a week. Then one evening a few extra stations come online, and the fleet collapses.
$ kubectl get pods -l app=tuner
NAME READY STATUS RESTARTS AGE
tuner-6b8f4d9c7-mn4kp 1/1 Running 7 (30s ago) 2d
tuner-6b8f4d9c7-qw8xn 0/1 Running 6 (12s ago) 2d
tuner-6b8f4d9c7-z4lr9 1/1 Running 8 (45s ago) 2dEvery pod restarting, repeatedly, within seconds of each other. No crashes in the logs. No OOM kills — Part 10 taught you to check, and the exit code is 137 but Reason says nothing about memory.
Nothing is wrong with the application. It is slightly slower than usual, because there is slightly more work. And something you installed to improve reliability is methodically killing every copy of it.
The three probes look nearly identical in YAML and do entirely different things. One controls traffic, one controls restarts, one controls the other two.
Then there's the other half of staying up, which nobody thinks about until a deploy drops requests: shutting down is a sequence, and it has a race in it.
Readiness is nearly free and nearly always right. It gates traffic and paces rollouts, and failing it costs you nothing but a pause.
Liveness can take down your whole fleet. It is the only probe that kills, and under load it kills everything at once. Add it when you can name the wedge it detects — not by default.
Click any piece of the command.
Click any piece of the command.
Click any piece of the command.
Readiness everywhere, a startup probe for the slow-loading speech model, a careful liveness probe, and a budget so node maintenance can't take the last one.
Probes
kubectl apply -f manifests/tuner-probes.yaml
kubectl rollout status deploy/tuner
kubectl describe pod -l app=tuner | grep -A3 -E 'Liveness|Readiness|Startup'Note the timings in that output. timeout=5s, not the default 1. period=10s #failure=3 — thirty seconds of genuine trouble before anything is killed, not one slow response.
A slow starter, handled properly
kubectl apply -f manifests/asr-startup.yaml
kubectl get pods -l app=asr -wIt sits 0/1 Running for a minute or so while the startup probe works through its budget, then flips to 1/1. No restarts, because liveness never ran. With only a liveness probe it would have been killed at twenty seconds and killed again every twenty seconds after that.
A disruption budget
kubectl apply -f manifests/asr-pdb.yaml
kubectl get pdbCheck ALLOWED DISRUPTIONS. If it reads 0, a node drain will block — which is either exactly right or a configuration you'll regret at the worst moment.
kubectl apply -f manifests/liveness-cascade.yaml
kubectl get pods -l app=cascade -wA health endpoint that takes about 1.2 seconds, and a liveness probe with the default timeoutSeconds: 1. Every pod is killed, comes back, is killed again — forever, while the application works perfectly throughout.
That single unset default is the whole bug. It is also the default you get by copying a probe snippet off the internet.
Fix the one number
kubectl patch deploy/cascade --type=json -p='[{"op":"replace",
"path":"/spec/template/spec/containers/0/livenessProbe/timeoutSeconds",
"value":5}]'
kubectl get pods -l app=cascade -wkubectl apply -f manifests/no-prestop.yaml
# hammer it from inside the cluster while you roll it
kubectl run load --rm -it --image=busybox --restart=Never -- \
sh -c 'while true; do wget -qO- -T2 http://no-prestop/ || echo FAIL; done'
# in another terminal
kubectl rollout restart deploy/no-prestopA scattering of FAIL lines. Those are requests that arrived at a pod which had already been told to stop — endpoint removal hadn't reached kube-proxy yet.
Now the version with a five-second preStop:
kubectl apply -f manifests/with-prestop.yaml
kubectl rollout restart deploy/with-prestopClean. The hook does nothing but hold the container open long enough for the rest of the cluster to stop sending it work. An embarrassing fix for a subtle problem.
kubectl apply -f manifests/strict-pdb.yaml # minAvailable == replicas
kubectl get pdb strict-pdb
kubectl drain airwaves-worker --ignore-daemonsets --delete-emptydir-dataThe drain refuses to evict, forever, and says why. A PDB that permits zero disruption is a permanent block on node maintenance — and you will find out during an urgent kernel patch.
kubectl uncordon airwaves-worker and delete the strict PDB.| Because… | …this is what happens |
|---|---|
| Readiness gates endpoints and rollouts | Without it, traffic and rollouts both assume “ready when started”. Two bugs, one omission. |
| Liveness restarts containers | Under load it fires on every pod at once and makes the problem worse. The failure amplifies. |
timeoutSeconds defaults to 1 | Far too tight for a busy process. The most consequential unset field in this part. |
| Startup probes gate the others | They let liveness stay strict without punishing slow boots. |
| Endpoint removal races shutdown | Requests arrive after SIGTERM. A preStop sleep is the standard fix. |
| PID 1 must handle SIGTERM | A shell wrapper swallows it, and every shutdown waits for SIGKILL instead. |
| PDBs only cover voluntary disruption | They block drains, never crashes. minAvailable == replicas blocks maintenance forever. |
• readiness = should I get traffic. liveness = should I be restarted. startup = have I finished booting.
• Readiness first, always. It gates Service endpoints and rollout pace. Failing it is safe.
• Liveness is the only one that kills. Add it when you can name the stuck state — not by default.
• timeoutSeconds defaults to 1. Raise it, or build yourself a cascade.
• Detection time is periodSeconds × failureThreshold, not periodSeconds.
• A startupProbe beats a long initialDelaySeconds — strict liveness, generous boot.
• Endpoint removal and SIGTERM happen at the same time. A preStop: sleep 5 closes the race.
• PID 1 must handle SIGTERM. A shell wrapper means everything waits for SIGKILL.
• terminationGracePeriodSeconds is the total budget, preStop included. Default 30.
• A PDB stops drains, never crashes. ALLOWED DISRUPTIONS: 0 blocks node maintenance.