Load Balancing: L4 vs L7, Algorithms, Health Checks

easy

Scale: 100K-1M conn/s per LB; <1ms added latency Storage: Connection table O(active conns) + backend health state Google, Cloudflare, Amazon

FundamentalsLoad BalancingNetworking

A load balancer spreads incoming traffic across a fleet of backend servers so no single one is overwhelmed, failures are bypassed, and the fleet can scale without client changes. The two layers — L4 and L7 — trade speed for routing intelligence.

Scale100K-1M conn/s per LB; <1ms added latency

StorageConnection table O(active conns) + backend health state

Key Concepts

1. L4 vs L7. L4 (HAProxy TCP, AWS NLB, IPVS) routes on connection metadata — IP, port, TCP options. Fast, protocol-agnostic, opaque. L7 (Nginx, Envoy, ALB) inspects the HTTP payload — path, headers, cookies — and can do TLS termination, retries, header rewrites, and per-route policies.

1. L4 vs L7.

2. Algorithm matters as much as layer. Round-robin is simple but blind to request cost. Least-connections is the strong default for general HTTP. Power-of-two-choices / EWMA (Envoy default) picks the better of two random backends — best p99 tail latency. Consistent hashing keeps the same key on the same backend — essential for cache fleets and sticky workloads.

2. Algorithm matters as much as layer.

3. Health checks make the LB useful. Active: LB probes /healthz periodically — catches slow drift. Passive: LB observes 5xx and connection errors on real traffic — catches fast spikes. Outlier detection ejects misbehaving backends with backoff. Graceful drain stops sending new connections, lets in-flight finish, then shuts down.

3. Health checks make the LB useful./healthz

4. Where the choices live. Stateless REST: L7 + least-connections or P2C. WebSocket / gRPC streams: L4 or L7 with sticky sessions (long-lived connections can't be rebalanced mid-flight). Memcached fleet: L7 with consistent hashing. Sticky sessions hurt elasticity — prefer stateless servers + an external session store (Redis).

4. Where the choices live.

5. In production. Cloudflare anycast at the edge + sophisticated L7 (HTTP/3, WAF). Netflix Zuul is an L7 edge LB with a rich filter chain. Envoy sidecars run L7 between services with mTLS and outlier detection. ALB / NLB are managed L7 / L4 respectively. Make the LB itself HA — single LB is a SPOF — via anycast, ECMP, or a hosted service.

5. In production.

Approach

Pick the layer. L4 if you don't need HTTP visibility, want maximum throughput, and tolerate opaque routing (NLB-style). L7 if you need path-based routing, header-aware retries, rate limiting, request transformations, or per-route TLS.
Pick the algorithm. Default to least-connections or P2C-EWMA for HTTP. Consistent hashing for cache fleets. Round-robin only when requests are truly uniform.
Health-check both ways. Active probes catch slow drift; passive observation catches fast spikes. Use outlier detection with backoff so a flaky pod doesn't oscillate.
Decide TLS strategy. Terminate at the LB for performance + observability. Re-encrypt to the backend if zero-trust or mTLS is required.
Make the LB itself HA. Pair of LBs behind a VIP, anycast announcement, or hosted (ALB / NLB / GCLB).
Wire up graceful drain on the deploy pipeline: stop new conns, let in-flight finish (deadline 30-60s), then terminate.
Instrument heavily — per-backend RPS, latency histogram, error rate, conn count. Dashboards on these catch 90% of LB issues.

Components

Frontend listeners — TLS terminator, HTTP/1.1/2/3 parsers, ALPN negotiation.
Backend pool — list of upstreams with per-target weight, health, and metadata.
Health checker — TCP, HTTP, gRPC health checks with thresholds.
Outlier detector — eject after N consecutive 5xx; probe back after backoff.
Routing rules — path / header / host-based selection of pool.
Per-route policies — retries, timeouts, circuit breakers, rate limits.
Configuration distribution — file reload, xDS (Envoy), API push.
Telemetry — access logs, metrics, distributed tracing headers.

Algorithms in depth

Round-robin: simple counter; uneven when request costs vary.
Weighted round-robin: assign capacity weights; static distribution.
Least-connections: routes to backend with fewest active conns. Good default.
Power-of-two-choices (P2C): pick two backends at random, route to less-loaded one. O(1), nearly optimal for tail latency.
EWMA / Maglev: exponentially weighted moving avg of recent latency; Envoy's default.
Consistent hashing: hash key → ring → backend. Same key → same backend even after scale-up. Used by cache LBs and stateful routing.
Random: surprisingly good with enough capacity headroom; no coordination.

Trade-offs

L4 vs L7: L4 cheaper CPU, lower latency, opaque routing. L7 richer behavior (retries, WAF, header rewrite) but doubles CPU per request.

Sticky vs stateless: sticky sessions hurt elasticity and concentrate failure. Stateless servers + external session store (Redis) is the default for a reason.

TLS at LB vs end-to-end: at LB simplifies certs and adds visibility, but the LB-backend hop is plaintext unless you re-encrypt. Required for some compliance regimes.

Active vs passive health checks: active probes catch drift but consume capacity. Passive catches reality but reacts to user-visible failures. Use both.

Consistent hashing vs modulo hashing: consistent moves only 1/N keys on rebalance vs ~all keys; modulo is simpler but devastating in cache fleets at scale.

Real-world patterns

Cloudflare: Anycast at the edge + sophisticated L7 in their network for HTTP/3, WAF, bot management.
Netflix Zuul: L7 LB at the edge with rich filter chain (auth, throttling, routing, decoration).
Envoy: per-pod sidecar in service meshes; service-to-service L7 with mTLS, retries, outlier detection.
AWS ALB/NLB: managed L7/L4 respectively; ALB does HTTP routing, NLB does TCP/TLS at higher throughput.
Google Maglev: software L4 LB with consistent hashing across a fleet, ECMP from routers.