Kubernetes by Example · Part 5
Three replicas gave you two pods on one machine and none on another. When the whole point is to be there, you need a different question.
You add the decoder to airwaves. It needs to be on every machine, because tuners hand it audio through shared memory and shared memory does not travel.
Three nodes, so you write a Deployment with three replicas. It comes up green: 3/3 ready.
And a third of your tuners are broken. Not crashing — just sitting there, waiting for a decoder that isn't on their machine.
$ kubectl get pods -o wide -l app=decoder
NAME READY STATUS NODE
decoder-x7k2p 1/1 Running node-1
decoder-m4n8q 1/1 Running node-1
decoder-p2w9s 1/1 Running node-2Two on node-1. One on node-2. Nothing at all on node-3.
The Deployment did precisely what you asked. You asked for three pods. You wanted one per node, and those are not the same request — there is no way to express the second one with a replica count.
Everything so far has been counted in replicas: a number you choose, pods the scheduler places wherever they fit. For most workloads that's exactly right — you care how much capacity exists, not where it sits.
Some workloads invert that completely. They don't have a sensible number; they have a place they must be.
Answer yes to any of these and you want a DaemonSet. Answer no to all three and you want a Deployment.
1. Does it need to exist wherever work might land? 2. Does it serve only clients on its own node? 3. Does it own a node-local resource — a device, a GPU, a shared memory region?
Click any piece of the command.
Click any piece of the command.
Click any piece of the command.
A multi-node cluster, then the decoder as a DaemonSet. If you've been running a single-node kind cluster, this is the point where you need more than one machine.
A cluster with three nodes
kind delete cluster --name airwaves
kind create cluster --name airwaves --config manifests/kind-3node.yaml
kubectl get nodesThe decoder, everywhere
kubectl apply -f manifests/decoder.yaml
kubectl get ds decoder
kubectl get pods -o wide -l app=decoderDESIRED is 3 and you never typed a 3. It was computed from the cluster. One pod per node, including the one the Deployment skipped.
Confirm the spread
kubectl get pods -o wide -l app=decoder --no-headers \
| awk '{print $7}' | sort | uniq -cOne per node, every node. That command is worth keeping — it's how you notice the machine that quietly has none.
kubectl taint nodes airwaves-worker2 maintenance=true:NoExecute
kubectl get pods -o wide -l app=decoder
kubectl get ds decoderThe pod on that node is evicted immediately — that's NoExecute rather than NoSchedule — and DESIRED drops from 3 to 2. The DaemonSet has not failed. It has correctly concluded that this node no longer wants it.
Now give it permission:
kubectl patch ds decoder --type=merge -p='
spec:
template:
spec:
tolerations:
- key: maintenance
operator: Exists
effect: NoExecute'
kubectl get pods -o wide -l app=decoderBack to three. The taint is still there — you didn't remove the node's objection, you gave the pod an exemption from it.
Put it back
kubectl taint nodes airwaves-worker2 maintenance=true:NoExecute-kubectl patch ds decoder --type=merge \
-p='{"spec":{"template":{"spec":{"nodeSelector":{"gpu":"true"}}}}}'
kubectl get ds decoderDESIRED 0, and every decoder pod gone. No error anywhere, because nothing is wrong — you asked for one per GPU node and there are no GPU nodes.
This is the DaemonSet failure mode worth recognising on sight. DESIRED 0 means your selector matched no nodes, and the fix is always on the node side:
kubectl get nodes --show-labels
kubectl label node airwaves-worker gpu=true
kubectl get ds decoderDESIRED climbs to 1 the moment the label exists. Nothing was restarted, nothing was reapplied — the controller simply ran its query again and found a different answer.
nodeSelector patch and the label when you're done experimenting.| Because… | …this is what happens |
|---|---|
| There is no replica count | You cannot scale a DaemonSet. The only way to have more is to have more nodes. |
| DESIRED is computed | DESIRED 0 means no node matched — look at node labels, not at the DaemonSet. |
| Nodes are tracked continuously | Scaling a node pool needs no change here. New nodes get the pod automatically. |
| There is no surge | Updates go node by node, old pod gone before new pod starts. Slow on purpose. |
| One pod serves one node | Its failure is bounded by node size — and bigger nodes mean a bigger blast radius. |
| Taints repel, tolerations permit | A toleration lets a pod land somewhere; it never attracts it there. |
• A Deployment counts pods. A DaemonSet counts nodes. Three replicas is not three machines.
• No replicas field exists. DESIRED is derived from how many nodes match.
• A node joins, a pod appears. No edit, no reapply — the reconcile loop over nodes.
• nodeSelector narrows which nodes qualify. DESIRED 0 means nothing matched, not that something broke.
• Nodes taint to repel; pods tolerate to be allowed. NoExecute also evicts what's already there.
• No surge on update — old pod must go before the new one starts. maxUnavailable counts nodes.
• Use one when locality is a requirement: node-local resources, local-only clients, must-be-everywhere agents.
• Don't use one just for even spreading — topologySpreadConstraints on a Deployment does that and keeps your count.
• A DaemonSet pod is a single point of failure for its node, and there is no replica count to raise.
• rollout restart ds/x respects pacing; delete pod -l does not.