Kubernetes by Example · Part 11
You gave the pod permission to use the GPU node, and it went somewhere else. Permission is not direction.
airwaves is getting a speech model, and it wants a GPU. One node has one. You add a toleration for the GPU taint, apply, and the pod schedules immediately.
Onto node-1. Which has no GPU.
$ kubectl get pods -o wide -l app=asr
NAME READY STATUS NODE
asr-5d8c9b4f7-qt2mx 0/1 CrashLoopBackOff node-1
$ kubectl logs deploy/asr --previous | tail -2
CUDA error: no CUDA-capable device is detectedYou gave it permission to use the GPU node and it went somewhere else entirely — which, once you see it, is exactly what you asked for.
A toleration is an exemption, not a destination. You told the scheduler the GPU node wouldn't reject your pod. You never told it to go there.
Three independent mechanisms, and most placement bugs come from using one where you needed two.
Pod-side rules pull toward nodes: nodeSelector, nodeAffinity, podAffinity. Node-side taints push everything away unless tolerated. Extended resources are simply counted, like CPU.
Taint the node so nothing else drifts onto your expensive GPUs.
Tolerate it on the pods that are allowed there.
And select or request so they actually go. The GPU request does this for you; a plain toleration does not.
Click any piece of the command.
Click any piece of the command.
Click any piece of the command.
Label a node, taint it, and put the speech model where it belongs — then pin a tuner beside the decoder it shares memory with.
Set the node up
kubectl label node airwaves-worker2 pool=gpu gpu=true
kubectl taint node airwaves-worker2 gpu=true:NoSchedule
kubectl get nodes -L pool,gpuThe model, with all three halves
kubectl apply -f manifests/asr.yaml
kubectl get pods -o wide -l app=asrThat manifest has a toleration and a nodeSelector. Remove either and it goes wrong in a different way — without the toleration it can't land, without the selector it lands anywhere.
Tuner beside its decoder
kubectl apply -f manifests/tuner-affinity.yaml
kubectl get pods -o wide -l 'app in (tuner,decoder)'Every tuner is on a node that has a decoder, because podAffinity with topologyKey: kubernetes.io/hostname made that a condition of scheduling. The shared memory from Part 9 only works because of this rule.
kubectl apply -f manifests/asr-toleration-only.yaml
kubectl get pods -o wide -l app=asr-wrongScheduled instantly, onto a node with no GPU. The toleration worked perfectly: it removed an obstacle. Obstacles removed are not directions given.
kubectl apply -f manifests/strict-antiaffinity.yaml
kubectl scale deploy/strict --replicas=4 # on a 3-node cluster
kubectl get pods -l app=strict
kubectl describe pod -l app=strict | tail -4Three running, one Pending forever. didn't match pod anti-affinity rules. You asked for replicas never to share a node, and there are only three nodes — so four replicas is arithmetically impossible.
This is fine on a quiet Tuesday and catastrophic during an incident, when you scale up for load and the extra pods silently refuse to exist. preferredDuringScheduling would have spread them and still allowed the fourth.
kubectl apply -f manifests/impossible.yaml
kubectl describe pod -l app=impossible | grep -A4 EventsThe numbers in that message total your node count, and each clause is a separate reason. Fix only the taint and the pods that lacked the label still won't take it.
kubectl taint node airwaves-worker2 gpu=true:NoSchedule- when you're done.| Because… | …this is what happens |
|---|---|
| A toleration is only permission | It lets a pod land somewhere; it never sends it there. Pair it with a selector. |
nodeSelector is a hard filter | No matching node means Pending forever — not an error, just an ungrantable wish. |
| Required means required | requiredDuringScheduling anti-affinity caps replicas at the node count. Prefer preferred. |
| Affinity is ignored during execution | Change a node's labels and running pods stay. Only new scheduling decisions see it. |
topologyKey defines togetherness | Hostname for same-machine (shared memory), zone for same-region resilience. |
| GPUs are whole and uncommittable | Request equals limit, integers only. One pod owns the device. |
| No device plugin, no resource | A node with a GPU that doesn't advertise one is invisible to the scheduler. |
• Default is best: let the scheduler choose unless you have a real reason not to.
• Pods pull (nodeSelector, affinity); nodes push (taints). Reserving hardware needs both.
• A toleration never attracts. It only removes an objection.
• required filters and can leave a pod Pending; preferred scores and always schedules somewhere.
• Required anti-affinity caps your replica count at your node count. Nearly always use preferred.
• topologySpreadConstraints is the proper tool for even distribution.
• topologyKey: kubernetes.io/hostname is how you demand same-machine co-location.
• Affinity is IgnoredDuringExecution — label changes never evict a running pod.
• GPUs are extended resources: whole units, request = limit, no overcommit.
• Read the FailedScheduling event. It counts every node and says why each refused.