Back to System design
Load Balancing: L4 vs L7, Algorithms, Health Checks
easyA load balancer sits in front of your servers and decides which one gets each request. It is also where a small mistake turns one broken server into a broken website.
Scale100K-1M conn/s per LB; <1ms added latency
StorageConnection table O(active conns) + backend health state
Key Concepts
1
Layer 4 forwards traffic; Layer 7 reads it. A Layer 4 balancer works at the network level. It picks a
server, forwards the connection, and does not look inside. It is fast and it works for any protocol, but it
cannot route by URL, retry a failed request, or handle HTTPS certificates.
Layer 4 forwards traffic; Layer 7 reads it.
2
A Layer 7 balancer reads the HTTP request. It can send
/api to one group and /images to another, retry
a request that failed safely, and terminate HTTPS. It costs more because it has to parse every request.
Most setups use both: Layer 4 at the front for volume, Layer 7 behind it for routing./api/images
3
The algorithm matters less than people think, with one exception.
The algorithm matters less than people think, with one exception.
4
Round robin hands out requests in turn. Fine when every request costs about the same.
5
Least connections sends to whichever server has the fewest requests in flight. Better when some requests
are slow, because a struggling server stops receiving new ones.
6
The exception worth knowing is power of two choices: pick two servers at random and send to the less
busy one. It gets almost all the benefit of tracking every server, without tracking any of them. That is
what you want when there are hundreds of servers and no single balancer sees the whole picture.
power of two choices
7
Health checks are where outages are born. The balancer asks each server "are you alive?" and stops
sending traffic to ones that do not answer.
Health checks are where outages are born.
8
Two things go wrong. A check that only says "the web server is running" passes while the application is
useless because the database is down. And a check that is too thorough becomes load in itself: 50
balancers checking 200 servers every second is 10,000 extra requests a second, arriving hardest when
things are already struggling.
9
The worst failure is a check that removes everything. A slow database makes every server fail its
check, so the balancer takes all of them out of rotation, and a slow website becomes no website.
The worst failure is a check that removes everything.
10
The protection is a rule: if fewer than about half the servers are healthy, ignore the health checks and
send traffic to everyone. A struggling server is better than no server. Proxies like Envoy call this panic
mode, and mentioning it shows you have thought past the happy path.
11
Draining is what makes deploys invisible. When you remove a server, stop sending it new requests but
let the ones in progress finish. Skip that and every deploy drops a few requests.
Draining is what makes deploys invisible.
12
What the interviewer is probing.
What the interviewer is probing.
13
1. "Layer 4 or Layer 7?" Probing: what each can do. Stalls: "Layer 7, it has more features."
Moves up: Layer 4 for raw speed and non-HTTP traffic, Layer 7 when you need routing, retries or HTTPS —
and both together is normal.
1. "Layer 4 or Layer 7?"
14
2. "Which algorithm?" Probing: matching it to the work. Stalls: "Round robin." Moves up: least
connections when request cost varies; power of two choices at large scale, and why it avoids shared state.
2. "Which algorithm?"
15
3. "How do you know a server is healthy?" Probing: how deep the check goes. Stalls: "It responds
to /health." Moves up: a check that touches a real dependency, weighed against what that costs across
the fleet.
3. "How do you know a server is healthy?"
16
4. "Your database slows down and every server fails its check." Probing: the cascade. Stalls:
"They are removed." All of them — now you serve nothing. Moves up: below a threshold, ignore the checks
and spread the load anyway.
4. "Your database slows down and every server fails its check."
17
5. "How do you deploy without dropping requests?" Probing: draining. Stalls: "Restart the
servers." Moves up: stop new requests, wait for the in-flight ones, then stop — and ramp a new server up
slowly rather than sending it full traffic cold.
5. "How do you deploy without dropping requests?"
Components
- Frontend listeners — TLS terminator, HTTP/1.1/2/3 parsers, ALPN negotiation.
- Backend pool — list of upstreams with per-target weight, health, and metadata.
- Health checker — TCP, HTTP, gRPC health checks with thresholds.
- Outlier detector — eject after N consecutive 5xx; probe back after backoff.
- Routing rules — path / header / host-based selection of pool.
- Per-route policies — retries, timeouts, circuit breakers, rate limits.
- Configuration distribution — file reload, xDS (Envoy), API push.
- Telemetry — access logs, metrics, distributed tracing headers.
Health check overhead, worked
Balancer instances 50 Backends 200 Check interval 1 second ------------------------------------------------------------- Checks per second 50 x 200 / 1 = 10,000 / second
If each check costs the backend 2 ms of work
Per backend 10,000 / 200 = 50 checks/s x 2 ms
= 10% of one core If the check touches the database
Database queries from checks alone = 10,000 / second
-- which can exceed real traffic on a quiet service, and
lands hardest exactly when the database is already sick. Cheaper shapes
- Balancers share health state (gossip or a control plane):
200 checks/s total instead of 10,000.
- Deep check on a slow interval (every 10 s) + shallow check
on a fast one (every 1 s).
- Passive checks: watch real responses; eject on consecutive
5xx, re-admit after a cooldown. Zero added load.Algorithms compared
Round robin
Even distribution of request COUNT.
Breaks when: request cost varies. One slow endpoint drags a
backend down while it keeps receiving its share. Least connections
Sends to the backend with fewest in-flight requests.
Breaks when: a backend fails fast -- errors return instantly, so
it looks idle and attracts more traffic. Pair with passive
ejection. Least response time (EWMA)
Tracks a moving average of latency per backend.
Breaks when: the average is stale after a backend recovers; needs
decay tuning. Power of two choices
Pick two at random, send to the less loaded.
Why it works: the probability of both samples being the worst
backend is small, so load evens out without global state.
Use when: hundreds of backends, many balancers, no shared view. Consistent hash (by key or session)
Same client or key lands on the same backend.
Use when: backend-local caches matter.
Breaks when: one key is hot -- you have pinned a hotspot.Failure modes in production
Health check cascade
Symptom: a slow dependency empties the entire pool.
Cause: every backend fails the same deep check simultaneously.
Fix: panic-mode threshold (serve to all backends when healthy
fraction drops below ~50%), and separate liveness from readiness.
Retry amplification
Symptom: a partial outage becomes total; load triples.
Cause: balancer retries plus client retries plus service-to-service
retries, multiplying at each hop.
Fix: retry budgets (cap retries as a fraction of traffic), jittered
backoff, and never retry non-idempotent requests blindly. Cold backend overwhelmed after deploy
Symptom: fresh instances return errors for the first seconds.
Cause: full traffic share sent to a JIT-cold, cache-empty process.
Fix: slow-start ramping over 30-60 s, and readiness that waits for
warm-up rather than for the port to open. Connection draining skipped
Symptom: a trickle of 502s during every deploy.
Cause: instances terminated while requests are in flight.
Fix: deregister, wait for in-flight to finish with a timeout, then
stop. The timeout must exceed your slowest normal request.