A load balancer is like the host at a busy restaurant. Guests arrive at one door, and the host sends each party to a table that is free. Nobody needs to know how many tables there are, or that one section has just closed for cleaning.
That analogy explains why load balancers are everywhere, and also why they are so often misunderstood. A host who keeps seating guests at a table with no waiter is worse than no host at all. Equally, spreading guests evenly across two tables in a section that has just flooded does not help anyone. A load balancer improves scale and availability only when the backends behind it, the health checks that judge them, and the failure domains they sit in are designed properly.
Hadeda Freight is fictional, and its address ranges and figures are illustrative. Hadeda's Track portal and API serve customers checking deliveries and drivers updating them from a mobile app, across a Johannesburg cloud region with three availability zones.
What a load balancer actually does
A load balancer presents one stable address for a service and distributes incoming work across a pool of healthy backends. In doing so it can provide several things at once:
- Scale, because more backends can be added behind the same address.
- Availability, because unhealthy backends stop receiving traffic.
- Safer change, because instances can be drained and replaced one at a time during deployments.
- A control point, because TLS, routing rules and a web application firewall can sit at the same place.
The first three only work if the backends are interchangeable. If Track stored a user's session in the memory of one app instance, every request from that user would have to go to that same instance, and losing it would sign them out. Keeping session state in a shared store (or in a signed token the client holds) is what lets the load balancer treat every instance as replaceable.
Layer 4 and Layer 7
Part 1 introduced the practical layers. Load balancers come in two families that map onto them.
| Type | Makes decisions using | Good fit |
|---|---|---|
| Layer 4 load balancer | IP addresses, protocol and port | TCP or UDP services, very high throughput, pass-through encryption |
| Layer 7 load balancer | HTTP host, path, headers, cookies and method | Websites and APIs, path routing, TLS termination, WAF integration |
A Layer 4 balancer moves connections. It sees that a TCP connection is arriving on port 5432 or a UDP datagram on port 514, picks a healthy backend, and passes it along without reading the content. That makes it fast and protocol-agnostic.
A Layer 7 balancer reads requests. Because it understands HTTP, it can route by meaning. Hadeda's API balancer has rules like these:
api.hadeda.example/shipments/*goes to the shipments service;api.hadeda.example/invoices/*goes to the billing service;- plain HTTP requests are redirected to HTTPS, or rejected for the API.
The trade-off is that Layer 7 must decrypt traffic to read it, and does more work per request. For Track's website and API that is exactly what is wanted. For the depot scanners' UDP telemetry, which has no HTTP request to read, a Layer 4 balancer is the right tool.
Inside Kubernetes the same choice appears with different names. A Service of type LoadBalancer is typically Layer 4; HTTP routing is handled by an Ingress or, increasingly, the Gateway API. At the time of writing (October 2026), the Kubernetes project recommends the Gateway API over the Ingress API, which is frozen though still supported, and the widely used ingress-nginx controller reached the end of its maintenance in March 2026. Teams still running it should plan a move.
Where TLS terminates
TLS encrypts traffic and, through certificates, proves to the client that it is talking to the real api.hadeda.example. The design question is where the encrypted connection ends, because whatever terminates TLS can see the traffic in the clear.
Common choices:
- At the edge or load balancer only. Simple and fast. Traffic between the load balancer and backends travels unencrypted inside the private network. Increasingly hard to justify for personal information.
- At the load balancer, then re-encrypted to the backend. The balancer can inspect and route, and the internal hop is still encrypted. This is Hadeda's choice for Track, since the traffic carries customer names, phone numbers and addresses.
- Passed through to the application. A Layer 4 balancer forwards encrypted traffic untouched, and the application terminates TLS. No inspection or path routing at the balancer.
- Mutual TLS between services, where both sides present certificates, so each service proves its identity to the other. Part 10 returns to this.
Whatever the design, certificates are operational assets. Each needs a named owner, automated renewal, secure private-key storage, and monitoring that alerts well before expiry. An expired certificate takes a service down just as completely as a failed server, and it is entirely predictable. It is also one of the failure tests in Part 11.
Health probes decide who gets traffic
The load balancer's judgement is only as good as its health probe. A TCP probe proves that a port opened. It does not prove the application can serve a request: a process can accept connections while its database pool is exhausted and every request fails.
A good health endpoint checks enough to make a sensible routing decision, without turning every minor dependency hiccup into a full outage. If Track's health check failed whenever the email provider was slow, a slow email provider would take every instance out of service at once, and the whole API would go down over a non-essential dependency. The health check should ask: "can this instance usefully serve requests right now?", not "is everything in the world fine?"
Where the platform supports it, separate two questions:
- Liveness: should this process be restarted? In Kubernetes, a failing liveness probe restarts the container.
- Readiness: should this instance receive new traffic? A failing readiness probe removes the pod from the Service's endpoints, so it receives no new requests, but leaves it running to recover.
Confusing the two causes restart loops: a liveness check that depends on the database will restart healthy containers every time the database blinks, which makes the recovery slower, not faster.
Zones, regions and global delivery
Track runs app instances in all three availability zones of its region, behind a regional load balancer that checks readiness. If one zone fails, its instances fail their probes, the balancer stops sending them traffic, and the remaining two zones carry the load. That only works if two zones can carry the full load, so Hadeda sizes each zone to handle half of peak traffic, not a third.
A global traffic service sits a level above. It can direct users to a healthy region based on latency, priority or geography, usually through DNS or anycast. A content delivery network caches suitable content close to users, which for Track means the portal's static files, not live shipment data.
Neither fixes an application that keeps session state on one server, or depends synchronously on a database that just failed. Global delivery gives you somewhere to send users; it does not make the place you send them work. For Hadeda, a second region is a recovery decision with data, latency and regulatory consequences, covered in Part 11.
Common mistakes
- Load balancing across instances in one zone and calling it resilient. The zone fails and takes every instance with it.
- TCP health checks for HTTP services. The port is open while every request returns errors.
- Health checks that depend on every dependency. One slow, non-essential service takes the whole fleet out.
- Liveness probes that check external dependencies. Healthy containers are restarted in a loop.
- Session state held on individual instances. Users are signed out whenever an instance is replaced.
- Certificates with no owner or renewal automation. An entirely predictable outage, scheduled by the expiry date.
- Unencrypted backend hops carrying personal information. Fine for a demo, hard to defend in a POPIA review.
Try it yourself
Choose Layer 4 or Layer 7 delivery for four Hadeda services: the public Track website, a private reporting database, the depot scanners' UDP telemetry collector, and the public API platform. For each, state where TLS terminates (if TLS applies at all), what the health check tests, and what happens to in-flight users when one availability zone fails.
Test your understanding
Why can a Layer 7 load balancer route /shipments and /invoices to different services, while a Layer 4 one cannot?
Answer: Layer 7 terminates TLS and reads the HTTP request, including its path. Layer 4 sees only addresses, protocol and port, so every request on port 443 looks the same.
An instance's TCP health probe passes, but users get errors. How?
Answer: The port accepts connections, but the application cannot serve requests, for example because its database connection pool is exhausted. A probe that exercises the application path would catch it.
What is the difference between a failing readiness probe and a failing liveness probe?
Answer: Readiness failure stops new traffic reaching the instance while it keeps running. Liveness failure restarts the container.
Track runs four instances behind a load balancer, all in Zone A. Is it zone-resilient?
Answer: No. Losing Zone A loses every instance. Resilience needs instances spread across zones, with each surviving zone able to carry the extra load.
What to take from this part
A load balancer is only as good as what sits behind it. Interchangeable backends, honest health checks and real failure domains.
Layer 4 moves connections; Layer 7 reads requests. Choose by what the service needs to understand.
Decide where TLS ends, deliberately. Whatever terminates it sees the traffic in the clear.
Readiness and liveness answer different questions. Mixing them turns a blip into a restart loop.
Spread across zones, and size each zone for the loss of another.
Next in the series: Connecting offices, data centres and cloud networks.
How CloudNala can help
We review application delivery paths end to end: which balancer type each service uses, where TLS terminates, what the health checks really test, and whether zone spread and capacity would survive losing a zone. For teams on Kubernetes, we help plan the move from frozen Ingress setups to the Gateway API without disturbing live traffic.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Request a Cloud Network Review or write to us at consult@cloudnala.co.za