Seeking Wisdom · AI/ML Engineering · Investing · Books · Projects · About ·

Kubernetes by Example · Part 7

Services and Ingress

Every component is healthy, and nothing can reach anything. The object that knows why is one most people never look at.

assumes Parts 1–6~18 minutesnetworking, finally

The symptom

The web page for airwaves is up. You create a Service in front of it, point something at the name, and get:

$ wget -qO- web:80
wget: can't connect to remote host: Connection refused

So you check the Service. It's fine.

$ kubectl get svc web
NAME   TYPE        CLUSTER-IP     PORT(S)   AGE
web    ClusterIP   10.96.41.7     80/TCP    4m

You check the pods. Also fine — Running, no restarts. You port-forward directly to a pod and the page loads perfectly.

Every component is healthy. The thing joining them is not, and nothing in either list will tell you so, because the object that knows is one you haven't met yet.

A stable front for an unstable set

By now the awkward fact is familiar: pods are disposable and their addresses expire. Everything that needs to reach one needs an intermediary — something with a name that holds still while the things behind it churn.

That's a Service. And like everything else in this series, it works by running a query.

Ingress airwaves.example.com routes by host and path, HTTP only Service · web web.airwaves.svc.cluster.local ClusterIP 10.96.41.7 — never changes selector app=web EndpointSlice — the live answer 10.244.1.17:8080 ready 10.244.2.31:8080 ready 10.244.1.44:8080 not ready — excluded rewritten every time a pod appears or leaves web-x2kmp 10.244.1.17 · app=web web-m9q4p 10.244.2.31 · app=web web-z4lr9 — starting has an IP, gets no traffic yet no Service at all tuner → decoder over /var/run/decoder.sock same node, by construction nothing to resolve, nothing to balance Host: → runs the query, keeps the answer kube-proxy picks one per connection empty here = connection refused, no error anywhere

The four types, briefly

ClusterIP — a virtual IP, reachable inside the cluster only. The default, and what most things want.

NodePort — plus a high port on every node. Crude but universal.

LoadBalancer — plus a real cloud load balancer. One per Service, billed accordingly.

Headless (clusterIP: None) — no virtual IP at all; DNS hands back the pod addresses.

The commands

Services, and the object that actually tells you what is wrong

Click any piece of the command.

Names, and testing them from inside

Click any piece of the command.

Ingress, and why yours does nothing

Click any piece of the command.

Build it

The web page behind a Service, the catalogue behind a headless one, and an Ingress in front of the lot.

A Service, and its real state

kubectl apply -f manifests/web-service.yaml
kubectl get svc web
kubectl get endpointslice -l kubernetes.io/service-name=web

Get into the habit of that second command now. The Service tells you almost nothing; the EndpointSlice tells you everything.

From inside the cluster

kubectl run probe --rm -it --image=busybox --restart=Never -- sh

# inside:
nslookup web
wget -qO- web:80
nslookup catalogue-0.catalogue
exit

Ingress

kubectl apply -f manifests/ingress.yaml
kubectl get ing
curl -H 'Host: airwaves.example.com' http://localhost/

On kind you need an ingress controller first — manifests/README.md has the one-liner. Without it the Ingress sits there with an empty ADDRESS forever, perfectly valid and entirely inert.

Break it

1 · Reproduce the symptom exactly

kubectl patch svc web --type=merge \
  -p='{"spec":{"selector":{"app":"web-ui"}}}'

kubectl get svc web                                      # looks perfect
kubectl get endpointslice -l kubernetes.io/service-name=web

The Service is untouched and healthy. The EndpointSlice is empty. One typo in a label and traffic goes nowhere, with no error at any layer — the Service has no opinion about whether its query should have matched something.

Diagnose it properly

kubectl describe svc web | grep -E 'Selector|Endpoints'
kubectl get pods -l app=web-ui          # nothing
kubectl get pods --show-labels -l app=web

Put it back

kubectl patch svc web --type=merge -p='{"spec":{"selector":{"app":"web"}}}'

2 · Watch readiness gate the endpoint list

kubectl scale deploy/web --replicas=4
kubectl get pods -l app=web -w
# in another terminal:
kubectl get endpointslice -l kubernetes.io/service-name=web -w

New pods get IPs immediately but do not appear as endpoints until they're ready. That lag is the entire reason a rolling update can be seamless — and the reason it isn't, when nobody wrote a readiness probe.

3 · See that port-forward proves nothing

kubectl patch svc web --type=merge -p='{"spec":{"selector":{"app":"nope"}}}'
kubectl port-forward svc/web 8080:80
curl localhost:8080      # works!

A completely broken Service, and port-forward is happy — because it resolves to a pod and tunnels there directly, skipping kube-proxy. Worth knowing before you use it as evidence that something works.

Put it back, and remember the order: Service → EndpointSlice → pods → labels. Not Service → pods.

What follows from this

Because……this is what happens
Services select by labelA typo matches nothing and reports success. Empty endpoints, no error.
Only ready pods are endpointsWithout a readiness probe, traffic arrives before the app can serve it.
The ClusterIP is virtualNo process owns it; kube-proxy rewrites in the kernel. Near-zero added latency.
Balancing is per connectionLong-lived gRPC or HTTP/2 pins to one backend. Headless or client-side balancing instead.
Headless returns pod addressesThe only way to address one specific replica. What StatefulSets need.
An Ingress is only a ruleWith no controller installed it does nothing, silently, forever.
port-forward skips kube-proxyIt proves the pod works, never that the Service does.

Cheat sheet

Part 7 · services and ingress

• A Service is a stable name and virtual IP in front of a changing set of pods.

• It finds backends with a label selector — the Part 3 mechanism again. No registration, no list.

• EndpointSlice is where you debug. Service healthy + endpoints empty = selector wrong or pods unready.

• Only ready pods become endpoints. No readiness probe means traffic arrives too early.

• kube-proxy rewrites in the kernel — no proxy hop, no real latency cost.

• Balancing is per connection. gRPC and HTTP/2 pin to one backend and stay.

• Headless (clusterIP: None) returns pod addresses directly. Required for StatefulSets.

• DNS: svc.ns.svc.cluster.local; bare name works in the same namespace only.

• An Ingress without a controller does nothing and says nothing. Check kubectl get ingressclass.

• Sometimes the right answer is no Service — a local socket to a DaemonSet needs no lookup and no balancing.