Seeking Wisdom · AI/ML Engineering · Investing · Books · Projects · About ·

Kubernetes by Example · Part 4

Deployments and Rollouts

You asked for a rolling update and got a gap anyway. Here is what actually moves during a deploy, and what sets its pace.

assumes Parts 1–3~17 minutesthe middle layer earns its keep

The symptom

You ship a new tuner image. The command returns instantly and cheerfully. Thirty seconds later someone asks why the transcripts stopped.

They stopped for about forty seconds, then came back. Nothing crashed. No pod shows a restart. kubectl get deployment says 4/4 ready and has said so the whole time, including during the gap.

You did a rolling update specifically to avoid this. It rolled. There was still a gap.

The answer is that a rolling update is only as careful as your definition of “ready”, and by default that definition is “the process has started”, which is not the same thing at all.

What a Deployment is actually for

If all you ever did was run N copies of one unchanging thing, you would not need a Deployment. A ReplicaSet does that by itself.

The Deployment exists for the moment you change your mind. It's a supervisor of versions: it keeps one ReplicaSet per version of your pod template and moves pods between them according to rules you set.

Deployment · tuner replicas: 4strategy: RollingUpdate ReplicaSet · tuner-7d4f8b revision 1 image: tuner:1.2 4 running, all ready ReplicaSet · tuner-9c2e1a revision 2 image: tuner:1.3 0 running — nothing has started yet one Deployment, many ReplicaSets a few at a time, never all at once the two numbers that set the pace maxUnavailable: 25%never fewer than 3 of 4 serving maxSurge: 25%never more than 5 of 4 existing new pods never become ready → it stops here …and the old ones keep serving. That is the feature.

Which edits cause a rollout

Inside spec.template — image, env, resources, labels, probes, volumes. Changes the template hash, so a new ReplicaSet appears and pods roll.

Outside it — replicas, strategy, revisionHistoryLimit. No new ReplicaSet, no rollout, takes effect immediately.

The commands

Driving and watching a rollout

Click any piece of the command.

Changing a Deployment without editing YAML

Click any piece of the command.

Counting, and pausing

Click any piece of the command.

Build it

Four tuners from Part 3, now with a rollout you can actually watch happen.

Apply and wait properly

kubectl apply -f manifests/tuner.yaml
kubectl rollout status deploy/tuner

Roll it

# in one terminal, watch pods come and go
kubectl get pods -l app=tuner -w

# in another, change the template
kubectl set env deploy/tuner TUNER_VERSION=2
kubectl rollout status deploy/tuner

In the watch window you'll see new pods appear before old ones leave — that's maxSurge — and never more than one missing at a time, which is maxUnavailable. Both ReplicaSets exist simultaneously throughout.

See both versions

kubectl get rs -l app=tuner
kubectl rollout history deploy/tuner

Two ReplicaSets: the old one at 0, the new one at 4. The old one is your rollback, sitting there costing nothing.

Break it

1 · Deploy something that cannot start

kubectl set image deploy/tuner tuner=busybox:does-not-exist
kubectl rollout status deploy/tuner --timeout=60s   # fails

kubectl get pods -l app=tuner
kubectl get rs -l app=tuner

Look carefully at that pod list. There's an ImagePullBackOff pod that will never start — and four healthy old pods still serving. The rollout took one step, couldn't take another, and stopped.

This is the system working. It will not trade a working fleet for a broken one just because you asked.

Undo it

kubectl rollout undo deploy/tuner
kubectl rollout status deploy/tuner

2 · Reproduce the gap from the top of the page

A container that takes ten seconds to be useful, and no readiness probe to say so.

kubectl apply -f manifests/slow-start.yaml
kubectl rollout status deploy/slow-start

# now roll it, and watch how fast every pod is replaced
kubectl set env deploy/slow-start V=2
kubectl get pods -l app=slow-start -w

Every pod is replaced in a couple of seconds, because the kubelet reports Running the moment the process exists and the Deployment believes it. For ten seconds afterwards, none of them can do anything useful. Your rollout reported complete success.

The fix is a readiness probe, which is Part 12. For now the lesson is narrower and more important: ready means whatever you define it to mean, and the default definition is nearly worthless.

What follows from this

Because……this is what happens
Deployments manage ReplicaSetsOld versions sit at zero, costing nothing, ready to be scaled back up. That is what makes rollback instant.
Template changes make new ReplicaSetsAny edit inside template triggers a rollout. Edits outside it don't.
Rollouts wait for readinessWithout a readiness probe, the pace is set by process startup, which is almost never the right moment.
A failed rollout stopsOld pods keep serving. A stuck rollout is protection, not breakage.
History is finiterevisionHistoryLimit bounds how far back you can undo. Keeping old ReplicaSets is free.
set edits live objectsConvenient during incidents, reverted by the next apply. Change the file for anything permanent.

Cheat sheet

Part 4 · deployments and rollouts

• A Deployment manages ReplicaSets; a ReplicaSet manages Pods. The middle layer is what makes change safe.

• Change anything in spec.template → new ReplicaSet → rollout. Change replicas → no rollout.

• maxSurge is how far above your count you'll go; maxUnavailable how far below. Both can't be zero.

• The rollout waits for readiness. No probe means “ready when the process exists”, which is usually a lie.

• A failed rollout stops and leaves the old pods serving. That's the feature, not the bug.

• rollout undo is fast because the old ReplicaSet never went away.

• rollout status is the command that tells you a deploy actually worked — apply never did.

• rollout restart rolls fresh pods without changing the image. Use it instead of deleting pods.

• rollout pause for a manual canary, or to batch several edits into one rollout.

• Recreate means downtime on purpose — for when two versions genuinely cannot coexist.