Kubernetes by Example · Part 2
It ran perfectly in Docker. You put it in a Pod and it died. Here is what a Pod actually is, and why that difference matters more than it looks.
You've written the actual tuner now. It works — you ran it locally with docker run and watched it print throughput numbers, happy as anything.
So you put it in the cluster. And:
NAME READY STATUS RESTARTS AGE
tuner-6b8f4d9c7-mn4kp 0/1 CrashLoopBackOff 4 2mFour restarts. You reach for the logs, obviously:
$ kubectl logs deploy/tuner
$Nothing. Not an error — nothing. An empty response and your prompt back.
At this point most people conclude that Kubernetes has eaten their logs. It hasn't. The logs are there, they're just attached to something that no longer exists, and finding them takes one extra flag that nobody tells you about on your first day.
To understand why, you need to know what you actually put your container inside.
In Part 1 we treated the Pod as a box with a container in it, which was fine because we only needed to watch it get created. Now we need to open the box.
A Pod is best thought of as a very small machine. It has one network address. It has a lifecycle — it starts, it runs, it ends, and it does not come back. It holds one or more containers that have agreed to be co-located, sharing some things and pointedly not sharing others.
Which things get shared, and which don't, explains most of the surprising behaviour you'll meet this week.
Shared automatically: the network — one IP, one localhost, one set of ports. And the lifecycle: they are placed together and they are deleted together.
Shared only if you ask: files, via a volume mounted into more than one container. Process visibility, via shareProcessNamespace.
Never shared: the root filesystem. Each container gets its own, from its own image.
Part 1 was about get and apply — commands for the list. These three are for the world: what a container said, what it's doing now, and what the cluster looks like from the inside.
Click any piece of the command.
Click any piece of the command.
Click any piece of the command.
A rough decision tree, since the hard part is usually knowing where to start rather than how to type it.
| The Pod is… | Start with | Because |
|---|---|---|
Pending | kubectl describe pod | It hasn't been placed. There are no logs yet because there is no container yet. The reason is in the events at the bottom. |
ImagePullBackOff | kubectl describe pod | Wrong name, wrong tag, or no credentials. Again, events — the message is usually explicit. |
Init:0/1 | kubectl logs -c <init> | Stuck in an init container. Name it explicitly or you'll be told the main one hasn't started. |
CrashLoopBackOff | kubectl logs --previous | It started and then died. The evidence belongs to the dead instance. |
Running but wrong | kubectl exec -it … -- sh | Go in and look. Check env, check mounts, check what it can reach. |
Running, no shell | kubectl debug | Attaches a container with tools, sharing the target's namespaces. |
Time to retire the placeholder. The real tuner reads a stream and reports how much audio it's getting — and it now has an init container that works out which stream, writing the answer to a volume the tuner reads back.
It runs entirely offline. By default the init container synthesises a local stream, so nothing depends on the internet or on somebody else's radio station staying up.
Apply it
kubectl apply -f manifests/tuner.yaml
kubectl get pods -l app=tuner -wWatch the STATUS column rather than waiting for it to settle. You should catch Init:0/1 go past — the init container doing its work — before Running appears. That transition is Part 2's whole thesis on screen for about half a second.
Read both containers
# the init container, which has already finished
kubectl logs deploy/tuner -c resolve-station
# the one still going
kubectl logs -f deploy/tuner -c tuner --tail=20Go in and look at what they share
kubectl exec -it deploy/tuner -c tuner -- sh
# inside:
cat /work/station.txt # written by a container that no longer exists
ip addr | grep inet # the Pod's IP, not the container's
env | grep STATION # configuration, from the manifest
exitThat cat is the point of the exercise. You are reading a file written by a different container, which has already exited, and it's there because the Pod declared a volume and both containers mounted it. Remove the volume and the file vanishes — not because it was deleted, but because the two containers were never looking at the same place.
Point the tuner at a station that doesn't exist and watch the symptom from the top of the page appear on demand.
kubectl set env deploy/tuner STATION_SOURCE=http://nowhere.invalid/stream
kubectl get pods -l app=tuner -w # wait for CrashLoopBackOff
kubectl logs deploy/tuner -c tuner # empty, or nearly
kubectl logs deploy/tuner -c tuner --previous # there it isThe first logs catches a container that has just been started and hasn't failed yet. The second reads the one that already died. Same Pod, same command, one flag apart — and the difference between twenty minutes of confusion and a ten-second fix.
While it's broken, look at how the cluster describes it:
kubectl describe pod -l app=tuner | grep -A6 'Last State'You get Terminated, a Reason, and an Exit Code. Exit code 1 means your program decided to quit. 137 means it was killed — almost always out of memory, which is Part 10. 143 means it was asked to stop politely, which is Part 12. Those three numbers cover most of what you'll meet.
kubectl set env deploy/tuner STATION_SOURCE=syntheticThis one is quick and the error message is wonderfully blunt.
kubectl apply -f manifests/port-clash.yaml
kubectl get pods -l demo=port-clash
kubectl logs -l demo=port-clash -c secondTwo containers, both trying to listen on 8080. The first wins, the second reports address already in use and dies. If you still half-believed containers in a Pod are isolated from each other, this is the demonstration that settles it.
kubectl delete -f manifests/port-clash.yaml| Because… | …this is what happens |
|---|---|
| A Pod has one IP | Containers inside reach each other on localhost with no config — and cannot both use the same port. |
| Containers don't share files | Anything passed between them needs a volume, declared on the Pod and mounted in both. |
| Init containers must succeed | A failing init container means the Pod never starts, which is the correct and much cheaper failure. |
| Restarts happen in place | The logs you need usually belong to the previous instance. --previous, every time. |
| Pods never move | Node problems are Pod problems. Nothing migrates; something new gets created elsewhere. |
| Pods are disposable | Never persist a Pod's name or IP. Both are temporary. Use labels and Services instead. |
Because containers in a Pod share so much, it's tempting to solve every problem by adding another container to the Pod. Resist it for now.
Two containers belong in the same Pod when they genuinely cannot work apart — they need the same filesystem, or the same network identity, or must be scheduled as one unit. If they'd work fine as separate Pods talking over the network, make them separate Pods. You get independent scaling, independent restarts and a smaller blast radius, all for free.
• The Pod is the unit of scheduling, not the container. One IP, one lifecycle, one node, forever.
• Shared free: network and lifecycle. Shared on request: files, via a volume. Never: the root filesystem.
• Init containers run first, one at a time, and must exit 0. Ordinary containers then start all together — there is no ordering between them.
• Two containers in one Pod cannot bind the same port. Same network namespace.
• A crash restarts the container in place, with growing backoff. That is what CrashLoopBackOff means.
• kubectl logs --previous is how you read the instance that died. Learn it before you need it.
• -c picks the container, init containers included.
• Exit 1 means it quit, 137 means it was killed (usually memory), 143 means it was asked to stop.
• kubectl run probe --rm -it --image=busybox --restart=Never -- sh for a look from inside the cluster.
• Pod names and IPs are temporary facts. Never write them down.