Seeking Wisdom · AI/ML Engineering · Investing · Books · Projects · About ·

Kubernetes by Example · Part 1

The Object Model

Why kubectl apply saying “created” tells you almost nothing — and what is really going on in the ten seconds afterwards.

starts from zero~15 minutesruns on a laptop

A bad first afternoon

It's your first afternoon with Kubernetes. Someone sends you a YAML file and a command to run. You run it. It says:

deployment.apps/tuner created

Created. Lovely. You type kubectl get pods to admire your work, and there it is, with a word next to it you weren't expecting:

NAME                    READY   STATUS    AGE
tuner-7d4f8b9c6-x2kmp   0/1     Pending   8s

You wait. You refresh. Still Pending. Nothing has gone red. Nothing has errored. And nothing is running.

Later the same afternoon you hit the mirror image of this. You delete a pod you don't want any more. It comes back. You delete it again, slightly more firmly this time. It comes back again, with a different name, looking entirely unbothered.

Both of these feel like the system is ignoring you. It isn't. It's doing exactly one thing, relentlessly and very literally, and neither behaviour makes the slightest sense until you can see what that one thing is. By the end of this page you'll be able to predict both.

Two words first

If you've never touched any of this, here are the only two bits of vocabulary you need before we start.

A container is a program plus everything it needs to run — its libraries, its config, its own little slice of filesystem — wrapped up so it behaves the same on your laptop as on someone else's server. If you've used Docker, you've used containers. If you haven't: think of it as a program you can hand to a colleague without also handing them three pages of installation instructions.

A cluster is a pile of computers that have agreed to be told what to do by one thing. It might be three machines in a rack, three hundred in a datacentre, or one laptop pretending to be several — which is what we'll use. Each machine in the pile is called a node.

Kubernetes is what tells the pile what to do. The interesting question is how, and the answer is odd enough to be worth the rest of this page.

The whole idea

Here it is. Everything else in this series — every one of the thirteen parts after this — is a consequence of this one sentence.

Kubernetes is a list of wishes and a crowd of pedants.

You write down what you want the world to look like. Not the steps to get there — just the destination. One tuner. This image. This much memory.

Then a crowd of small, extremely literal programs read your list, look at the world, and notice differences. Each one knows how to fix exactly one kind of difference and nothing else whatsoever. It makes its one small correction. Then it reads the list again. And again. Forever, until somebody turns the cluster off.

None of them talk to each other. They genuinely can't — there is no channel between them. The only thing any of them can do is read the list and scribble on it.

So when you run kubectl apply, you are not issuing a command. There is no command. You are editing a list. Everything that happens afterwards happens because somebody noticed that the list had changed and the world hadn't caught up.

Walking it through, once, slowly

Let's follow a single wish from your keyboard to a running process. Nine steps, and only one of them involves anything actually happening.

you kubectl apply -f … the only door api server checks it · writes it · tells everyone etcd the list itself the pedants deployment controller makes ReplicaSets replicaset controller makes Pods, counts them scheduler writes down a node name a node — an actual computer kubelet anything on the list for me? Pod · tuner container image pulled · process alive write it down read · fix one thing · read again anything for me? start it gone — so the list is wrong again

Three words, now that you've seen them work

Spec is the wish — what you asked for. Status is the world — what's actually true. Every object in Kubernetes carries both, side by side, and you can read them with kubectl get pod X -o yaml.

Reconciliation is the name for the gap-closing. It's a loop, not an event. That's why the system recovers from failures nobody planned for — and equally, why it will cheerfully undo the clever manual fix you just made.

The anatomy of a kubectl command

Before we build anything, it's worth taking kubectl apart. Almost every command you will ever type has the same four-part shape, and once you can see the shape you can build complicated commands by assembling pieces rather than by hunting for someone else's one-liner.

kubectl  <verb>  <type>[/<name>]  [flags]
         │       │                  │
         │       │                  └─ narrow it, format it, point it somewhere
         │       └─ what kind of thing, and optionally which one
         └─ what to do: get, describe, apply, delete, logs, exec…

That's it. Three commands below, pulled apart piece by piece — click any part to see what it does and how far you can push it.

The command that starts everything

Click any piece of the command.

The command you will type ten thousand times

Click any piece of the command.

Pulling one value out, for scripts

Click any piece of the command.

The flags that work almost everywhere

These compose with nearly every verb, which is what makes them worth memorising rather than looking up.

FlagWhat it doesThe reason you'll reach for it
-n <ns>Work in one namespaceWithout it you're in default and wondering where everything went. -A means every namespace at once.
-l <sel>Filter by labelThe main way to say “all the pods belonging to this thing”.
-o <fmt>Change outputwide to read, yaml to understand, jsonpath to script, name to pipe.
-wWatch, don't exitStreams changes as they happen. The nicest way to actually see reconciliation occur.
--dry-run=client -o yamlBuild, don't sendGenerates correct YAML without touching the cluster. The fastest way to write a manifest you haven't memorised.
--context <c>Pick the clusterCheap insurance against the single most expensive mistake in this whole field.
The trick worth stealing: --dry-run=client -o yaml turns kubectl into a YAML generator. kubectl create deployment tuner --image=busybox --dry-run=client -o yaml > tuner.yaml gives you a correct, complete starting point in one line, and you edit from there instead of from a blank file.

Build it

A one-node cluster and the smallest tuner that can possibly exist. The container here is a placeholder that prints a line every second — we're watching the machinery, not the workload. The real one arrives in Part 2.

Make a cluster out of your laptop

kind create cluster --name airwaves
kubectl cluster-info --context kind-airwaves

Write the wish down

kubectl apply -f manifests/tuner.yaml

# read back what the cluster now believes
kubectl get deployment tuner -o yaml | head -40

See the chain you just stepped through

# all three objects, in the order they were created
kubectl get deployment,replicaset,pod -l app=tuner

# who made this pod?
kubectl get pod -l app=tuner \
  -o jsonpath='{.items[0].metadata.ownerReferences[0].kind}{"\n"}'

# the pedants, narrating themselves
kubectl get events --sort-by=.lastTimestamp | tail -12

That last one is the command to keep. The event stream is the controllers thinking out loud — Scheduled, then Pulling, then Created, then Started — and every line was written by a different program that noticed a different difference. You just watched the loop from the outside.

Break it

Two experiments. The first shows the loop working for you. The second shows it working against you, which is by far the more useful half.

1 · Delete the pod

kubectl delete pod -l app=tuner
kubectl get pods -l app=tuner -w

A new pod shows up within a second or two, wearing a different random suffix. It is not your pod restarted. It is a different pod, created from scratch, because a program that can only count noticed that one did not equal zero.

2 · Ask for something impossible

Now let's reproduce the Pending from the top of the page on purpose.

# demand more CPU than any node could ever have
kubectl patch deployment tuner --type=json \
  -p='[{"op":"replace",
        "path":"/spec/template/spec/containers/0/resources/requests/cpu",
        "value":"64"}]'

kubectl get pods -l app=tuner
kubectl describe pod -l app=tuner | tail -8

The new pod sits at Pending and stays there, and describe ends with FailedScheduling. Nothing errored. Your patch succeeded — the wish was recorded perfectly — and the scheduler is doing its job correctly by refusing to pretend something fits when it doesn't.

That's the mystery from your first afternoon, solved. The apply was never the problem. The wish simply can't be granted, and the system will keep patiently trying to grant it until the heat death of the universe or you fix the list, whichever comes first.

Put it back: run the same patch with "value":"50m" and watch the Pending pod get replaced by a running one without you touching anything else. You changed the list; the pedants did the rest.

What follows from all this

Four consequences that fall straight out of the model, and that most people learn one expensive afternoon at a time.

Because……this is what happens
The list always winsHand-editing something a controller owns gets quietly reverted. Edit the thing that asks for it, not the thing you can see.
Pedants never talk to each otherThere's no transaction spanning objects. A half-built state isn't a bug, it's Tuesday. Give it a few seconds.
apply only writesA green command is not a working deployment. If you need to know, ask separately: kubectl rollout status deploy/tuner.
Reconciling is a loopDeleting something you don't want is usually the wrong move. Change what's asking for it, and let the loop clean up.

Cheat sheet

Part 1 · the object model

• Kubernetes stores what you want and works toward it continuously. It does not execute your commands.

• The API server is the only door. Controllers, scheduler and kubelet all talk to it and never to each other.

• Spec is the wish, status is the world, and the gap between them is what everything here exists to close.

• A Deployment makes a ReplicaSet, which makes Pods. Three objects, three controllers, three differences.

• The scheduler only writes down a node name. The kubelet is the one that actually starts a container.

• apply returning success means recorded. Never running.

• Delete something a controller owns and it comes back. Change the owner instead.

• Every command is kubectl <verb> <type> [flags]. Learn the shape once, build the rest.

• An idle cluster is a healthy one — every pedant looking, and finding nothing to fix.