Kubernetes by Example · Part 13
An autoscaler that will never scale, and a pod quietly forbidden from doing its job. Both report perfect health.
Two problems, both of which look like nothing happening.
The first: somebody queues two thousand hours of archive for re-transcription. The backlog climbs. The rewind worker sits at one replica, serenely.
$ kubectl get hpa rewind
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS
rewind Deployment/rewind /70% 1 10 1 The second: the sweeper is meant to clean up shared memory for departed stations. It isn't. Its logs are clean. The pod is healthy. The rings pile up until the node runs out.
$ kubectl logs deploy/sweeper | tail -2
sweeper: starting
sweeper: listing pods on node airwaves-workerIt says it's listing pods, and then says nothing ever again.
One of these is an autoscaler that cannot see its metric. The other is a pod that is not allowed to ask the question it's asking — and neither failure is reported anywhere you'd think to look.
These two topics sit together because they're both about a pod's relationship with the cluster rather than with its own work. One changes how many of you there are. The other decides what you may ask.
And both fail silently in the same irritating way: the system reports perfect health while doing nothing at all.
TARGETS: <unknown> — no metrics-server, or no CPU request on the pod. The HPA exists, is healthy, and will never act.
A 403 in your pod's logs and nowhere else. No event, no status, no describe output. kubectl auth can-i answers it instantly.
Click any piece of the command.
Click any piece of the command.
Click any piece of the command.
The web front end scaling on CPU, the rewind worker scaling on backlog down to zero, and the sweeper finally allowed to do its job.
CPU scaling
kubectl apply -f manifests/metrics-server.yaml # kind needs this
kubectl top nodes # confirm it works
kubectl apply -f manifests/web-hpa.yaml
kubectl get hpa webIf TARGETS shows a percentage, you're fine. If it shows <unknown>, stop here — nothing below will work, and the cause is one of exactly two things.
Queue scaling, down to zero
kubectl apply -f manifests/keda.yaml
kubectl apply -f manifests/rewind-scaledobject.yaml
kubectl get scaledobject,hpa,pods -l app=rewindNo rewind pods at all, and that's correct — the queue is empty. KEDA created an HPA underneath; the mechanism is the same, only the metric is better.
Permission for the sweeper
kubectl apply -f manifests/sweeper-rbac.yaml
kubectl auth can-i list pods \
--as=system:serviceaccount:airwaves:sweeper -n airwavesyes. Three objects — ServiceAccount, Role, RoleBinding — and the pod can finally ask its question. Open sweeper-rbac.yaml and notice how narrow the Role is: list and watch on pods, nothing else. It needs nothing else.
kubectl apply -f manifests/no-requests.yaml
kubectl autoscale deploy/no-requests --min=1 --max=5 --cpu-percent=70
sleep 60
kubectl get hpa no-requests
kubectl describe hpa no-requests | tail -5<unknown>, and the events say missing request for cpu. The HPA is perfectly healthy and structurally incapable of acting, because 70% of an unspecified request is not a number.
Add a request and it starts working within a minute, with no other change.
kubectl delete rolebinding sweeper
kubectl logs -f deploy/sweeperIf you're lucky the client logs the 403. If it swallows errors — and plenty do — you get silence, which is the symptom from the top of this page.
Confirm it properly instead of guessing:
kubectl auth can-i list pods \
--as=system:serviceaccount:airwaves:sweeper -n airwaves # no
kubectl auth can-i --list \
--as=system:serviceaccount:airwaves:sweeper -n airwavesTwo seconds, and unambiguous. The ServiceAccount still exists, the Role still exists, and without the binding between them the pod can do nothing — which is precisely the shape of the most common RBAC mistake.
kubectl auth can-i --list \
--as=system:serviceaccount:airwaves:default -n airwavesOn a stock cluster this is a short and reassuring list. On one where somebody once bound cluster-admin to the default account to make something work, it is not — and every pod in the namespace inherits it, including anything that ever gets compromised.
sweeper-rbac.yaml.| Because… | …this is what happens |
|---|---|
An HPA just writes replicas | Your apply fights it. Remove replicas from manifests an HPA owns. |
| CPU targets are % of request | No request means no scaling, ever. TARGETS: <unknown>. |
| Scale-down is stabilised | Up is fast, down waits five minutes. Deliberate, and tunable via behavior. |
| HPAs stop at 1 | Scale-to-zero needs KEDA or similar — and CPU was the wrong metric for a queue worker anyway. |
| Zero has a cold start | The first message waits for a pod to boot. Keep one warm if a human is waiting. |
| A ServiceAccount grants nothing | It is identity only. Role + RoleBinding + ServiceAccount, and the binding is the one forgotten. |
| 403s are returned, not recorded | No event, no status. kubectl auth can-i --as=… is the diagnostic. |
• An HPA is a controller that writes replicas. Everything downstream is Part 1's loop.
• CPU targets are a percentage of the request. No request → <unknown> → never scales.
• Needs metrics-server. Not installed by default on kind.
• Up is quick; down waits ~5 minutes. Tune with behavior.
• Remove replicas from any manifest an HPA owns, or they will fight.
• HPAs can't reach zero. KEDA can — and queue depth beats CPU for workers anyway.
• Scale-to-zero costs a cold start. minReplicaCount: 1 buys it back.
• Every pod runs as a ServiceAccount; by itself it can do nothing, which is correct.
• Three objects: ServiceAccount + Role + RoleBinding. The binding is the one people forget.
• kubectl auth can-i --as=system:serviceaccount:ns:name before debugging any application code.