Kubernetes by Example · Part 3
Nothing in Kubernetes finds anything by name. Once you know what it does instead, half the surprising behaviour stops being surprising.
You scale the tuner up to four, because four stations is more interesting than one. It works.
Then you make a second Deployment for a different kind of tuner, by copying the first manifest and changing the name. And now something is very wrong: pods are being deleted. Not by you. Your four tuners become three, then five, then two. Neither Deployment ever reaches the number it's asking for.
Nothing has errored. Both Deployments report themselves as perfectly healthy. The logs say nothing, because from each pod's point of view nothing unusual is happening — it's simply being killed, over and over, by something that won't explain itself.
What you've built is two owners for the same set of pods. To see why that's even possible, you need to know how anything in Kubernetes finds anything else.
This is the part that feels wrong at first and then becomes the most useful thing you know.
A ReplicaSet does not hold a list of the pods it made. A Service does not hold a list of its backends. They hold a query, and they run it, constantly, and act on whatever it returns.
Which means the relationships you draw on a whiteboard — this Deployment owns those pods, that Service fronts these — don't exist as links anywhere. They're re-derived from scratch, several times a second, from labels you wrote by hand.
Labels are indexed and queryable. Short, restricted alphabet, 63 characters. Use them for anything that needs finding.
Annotations are not queryable at all. Any size, any content. Use them for things that are read after an object has been found — checksums, descriptions, config for other tools.
Selectors are the most reusable thing in kubectl. Learn them here and they work on every verb for the rest of the series.
Click any piece of the command.
Click any piece of the command.
Click any piece of the command.
Four tuners, one per station, distinguished by labels rather than by name. This is the shape the rest of the series builds on: identical pods carrying different tags.
Apply, and show the station as a column
kubectl apply -f manifests/tuners.yaml
kubectl get pods -L station -l app=tunerAsk it questions
# every tuner
kubectl get pods -l app=tuner
# only the jazz ones
kubectl get pods -l 'app=tuner,station=jazz'
# everything except the test station
kubectl get pods -l 'app=tuner,station notin (test)'
# anything that forgot to declare a station
kubectl get pods -l 'app=tuner,!station'That last one is the habit worth forming. A selector for the absence of a label is how you find the thing someone forgot, and it costs nothing to run.
Create a pod that has nothing to do with your Deployment, but happens to wear the right label.
kubectl run stray --image=busybox --restart=Never \
--labels='app=tuner' -- sleep 3600
kubectl get pods -l app=tuner -wWatch what happens. The ReplicaSet runs its query, finds one more pod than it asked for, and deletes one — possibly your stray, possibly one of its own. It has no way to tell the difference, because there is no difference. Ownership here is just a matching label.
kubectl patch deployment tuner --type=merge \
-p='{"spec":{"selector":{"matchLabels":{"app":"tuner","v":"2"}}}}'Rejected, with field is immutable. Now you know why: changing the query would orphan every pod the Deployment currently owns, and Kubernetes has no mechanism to hand them over or clean them up. It refuses rather than leaving you with ghosts.
# take it out of the Service's query without killing it
kubectl label pod station=quarantine --overwrite
kubectl get pods -L station -l app=tuner The pod keeps running. The ReplicaSet still owns it, because app=tuner is unchanged. But a Service selecting station=jazz drops it instantly, which means you can take a misbehaving pod out of rotation and then go poke at it at your leisure.
| Because… | …this is what happens |
|---|---|
| Ownership is a query | A ReplicaSet adopts any unowned pod that matches. Hand-made test pods get eaten. |
| Selectors are immutable | Pick a minimal one — app=x — and put everything volatile in the template labels. |
| Overlaps aren't validated | Two Deployments can fight over one pod forever, silently. Change the selector when you copy a manifest. |
| A typo matches nothing | A wrong selector doesn't error, it returns zero. Empty results mean “check the query” before “check the pods”. |
| Labels move freely | Relabelling is instant rerouting. Canaries, quarantine and blue/green are all one kubectl label. |
| Namespaces scope names only | Different namespaces can't collide on names and can't select each other — but they can still talk over the network. |
• Labels are tags you invent. Kubernetes gives them no meaning; selectors do.
• Nothing finds anything by name. Owners and Services run queries, constantly, and act on the results.
• Equality (a=b, comma = AND) and set-based (in, notin, key, !key). No OR across keys, no wildcards.
• A ReplicaSet adopts any matching unowned pod — including ones you made by hand.
• A Deployment's selector is immutable. Keep it minimal; put the changing labels in the template.
• Overlapping selectors across two workloads cause silent, permanent pod churn.
• Labels are indexed and queryable; annotations are neither, but can be any size.
• -l '!key' finds everything missing a label — the cheapest audit there is.
• Relabel a pod to pull it out of a Service without killing it.
• Namespaces scope names. They are not a security or network boundary by themselves.