Kubernetes by Example · Part 10
A pod killed with no error, and a node that is full at ten percent. Both are the same misunderstanding.
Two complaints arrive the same week, and they look unrelated.
The first: a tuner keeps dying. No crash in the logs, nothing wrong with the stream, and the pod restarts cleanly each time.
$ kubectl describe pod tuner-6b8f4d9c7-mn4kp | grep -A3 'Last State'
Last State: Terminated
Reason: OOMKilled
Exit Code: 137The second: you try to add a fifth tuner and it will not schedule.
$ kubectl describe pod tuner-6b8f4d9c7-x9k2w | tail -3
Warning FailedScheduling 0/3 nodes are available:
3 Insufficient cpu.
$ kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
node-1 412m 10% 1894Mi 24%Insufficient CPU on a node running at 10%.
Both of these are the same misunderstanding wearing different clothes. There are three different numbers people mean by “how much CPU this uses”, and Kubernetes cares about a different one than you probably do.
What it uses is a measurement — real, observed, changing every second. What it requested is a reservation the scheduler honours. What it's limited to is a ceiling the kernel enforces.
Only the second one decides whether your pod fits anywhere. Only the third one decides whether it gets killed. The first decides neither.
CPU is compressible. Over the limit, you get throttled — slower, still alive, and completely silent. No event, no restart, nothing in kubectl.
Memory is not. Over the limit, you get killed. Immediately, exit 137, no grace.
Which is why a too-low memory limit announces itself and a too-low CPU limit hides for months.
Click any piece of the command.
Click any piece of the command.
Click any piece of the command.
Measure first, then set numbers that mean something. Guessing is how you get both of this page's symptoms.
What is it actually using?
kubectl top pods -l app=tuner --containers
kubectl top pods -A --sort-by=memory | headSet them, and see the class you earned
kubectl apply -f manifests/tuner-resources.yaml
kubectl rollout status deploy/tuner
kubectl get pod -l app=tuner \
-o jsonpath='{.items[0].status.qosClass}{"\n"}'The number the scheduler reads
kubectl describe node | grep -A8 'Allocated resources'Compare that table with kubectl top nodes. The percentages will disagree, often wildly, and the one in describe is the one that decides whether anything else can land here.
kubectl apply -f manifests/memory-hog.yaml
sleep 20
kubectl get pods -l app=memory-hog
kubectl describe pod -l app=memory-hog | grep -A3 'Last State'OOMKilled, exit 137, and a restart count climbing. The container asked for more memory than its limit and the kernel ended the conversation. Nothing negotiated, nothing logged by the application — it simply stopped existing mid-instruction.
kubectl apply -f manifests/cpu-throttle.yaml
sleep 30
kubectl get pods -l app=cpu-throttle # Running, 0 restarts, perfectly healthy
kubectl logs deploy/cpu-throttle | tail -5The pod is fine. Kubernetes says it's fine. The logs show work taking several times longer than it should, and no object anywhere records that anything is wrong.
This is the failure mode to fear. A memory limit that's too low wakes you up; a CPU limit that's too low just quietly makes everything worse until someone complains about latency months later.
kubectl apply -f manifests/greedy-request.yaml
kubectl get pods -l app=greedy
kubectl describe pod -l app=greedy | tail -4
kubectl top nodesInsufficient cpu while the node is nearly idle. The pod requested more than any node has left in reservations, and actual usage was never part of the calculation.
This is also Part 1's experiment, now with a proper explanation: the wish is recorded faithfully and can never be granted, so the loop keeps trying forever.
| Because… | …this is what happens |
|---|---|
| The scheduler reads requests only | An idle node can be unschedulable; a busy one can have room. top doesn't answer this question. |
| No requests means zero | The pod fits everywhere, backs nothing, and is evicted first under pressure. |
| CPU throttling is silent | Latency with no event, no restart, no status change. Only metrics show it. |
| Memory kills are immediate | Exit 137, no grace period, nothing flushed. The kernel, not Kubernetes. |
| Limits can be overcommitted | Normal and useful — until everything bursts at once. |
| QoS is derived | You never set it. Requests and limits decide your eviction order whether you meant them to or not. |
| LimitRange can inject defaults | A pod may have numbers you never wrote. Check before assuming your YAML is in force. |
• Three different numbers: used (observed), requested (reserved), limited (enforced).
• The scheduler reads requests and nothing else. Not usage, not limits.
• No requests = zero requested. Fits anywhere, evicted first.
• CPU over limit → throttled, silently. No event, no restart. Only cfs_throttled_seconds shows it.
• Memory over limit → killed instantly. Exit 137, OOMKilled, no grace.
• Limits may be overcommitted; requests may not. The gap is how much you're betting.
• QoS is derived: Guaranteed (req=lim), Burstable (req<lim), BestEffort (nothing). It sets eviction order.
• CPU is millicores (1000m = 1 core). Memory: Mi is 1024-based, M is 1000-based.
• Allocatable < capacity. The kubelet and OS take their cut first.
• Measure, then set. Memory request ≈ limit. Be careful with CPU limits — many teams omit them deliberately.