resilience
Retries, Timeouts & Backoff
Recover from transient failures via retry — but never without a timeout, exponential backoff, jitter, and an attempt cap.
Networks fail transiently all the time — a dropped packet, a momentary GC pause, a brief blip during a deploy — and many of these failures succeed if simply tried again. Retrying is therefore one of the most valuable resilience tactics, but it is also one of the most dangerous when done naively, because careless retries can amplify a small problem into a self-inflicted outage. The full pattern is retries done with timeouts, backoff, jitter, and a hard cap.
Knocking on a door: if no answer you wait a bit longer each time rather than pounding continuously, and you give up after a few tries instead of standing there forever.
Key Concepts
1
The pieces work together. A timeout is non-negotiable and comes first: without one, a call to a hung dependency waits forever, holding a thread and a connection, so every request must have a bounded deadline. Retries then re-attempt a failed call, but immediate retries make things worse — if a service is overloaded, retrying instantly just doubles the load. Exponential backoff spaces attempts out by an increasing delay (1s, 2s, 4s), giving the dependency time to recover. Jitter adds randomness to those delays so that many clients failing at the same instant do not all retry in lockstep and create a synchronized "thundering herd" spike — this is the detail people most often omit. And a maximum attempt cap ensures retries eventually give up rather than hammering forever. Critically, you should only retry idempotent operations, or non-idempotent ones guarded by an idempotency key, because retrying a non-idempotent POST that actually succeeded but whose response was lost will duplicate the side effect.
2
The trap interviewers love is retry amplification in a call chain: if service A retries B three times, and B retries C three times, a single failure at C can become nine requests, multiplying load exactly when the system is already struggling. The defences are capping retries, retrying at only one layer, distinguishing retryable failures (timeouts, 503, 429) from non-retryable ones (400, 404), and combining retries with a circuit breaker so a persistently failing dependency stops being retried at all.