Reliability: Retries, Breakers, Backpressure, Bulkheads

The failure mode that defines the module

A normal overload is self-correcting: remove the spike and the system recovers. A metastable failure is not. A trigger (a blip, a deploy, a traffic spike) pushes the system into a state that sustains itself through a feedback loop — almost always retries — even after the trigger is gone. The defining test: if removing the original cause does not restore service, you’re in a metastable failure, and you cannot wait it out. Drive the simulator below into one.

The Resilience Stack

Here is the full stack, in the order a request encounters it. Read it as a defense-in-depth diagram: each layer catches a failure the layers above let through — and each, if misconfigured, manufactures the failure in the right-hand column. That second column is the part nobody teaches and every incident review rediscovers.

The Retry Budget: the one defense that stops the storm

Of all these layers, one is non-negotiable for preventing metastable collapse: the Retry Budget. Backoff and jitter spread retries out in time, but they don’t bound the total retry volume — and under a sustained partial outage, even well-spaced retries add up to a storm. A retry budget caps retries as a fraction of total traffic (say, 10%): retries draw tokens from a shared bucket that refills with successful traffic, and when the bucket is empty, you stop retrying and fail fast. That cap is what breaks the feedback loop.

1type RetryBudget struct {
2  tokens, max, depositPerCall float64
3}
4 
5// Each top-level call deposits a little (e.g. 0.1 token/call -> retries
6// capped at ~10% of traffic). A healthy system refills the budget faster
7// than retries drain it; a struggling one runs the budget dry and STOPS
8// retrying, which is exactly what breaks the metastable loop.
9func (b *RetryBudget) Deposit() {
10  b.tokens += b.depositPerCall
11  if b.tokens > b.max { b.tokens = b.max }
12}
13 
14func (b *RetryBudget) TryRetry() bool {
15  if b.tokens >= 1 { b.tokens--; return true }
16  return false // budget exhausted -> do not retry, fail fast
17}

What this sounds like in an interview

Exercises

1async function withRetry<T>(fn: () => Promise<T>, attempts = 3): Promise<T> {
2  let lastErr: unknown
3  for (let i = 0; i < attempts; i++) {
4    try {
5      return await fn()                 // e.g. () => paymentGateway.charge(...)
6    } catch (err) {
7      lastErr = err                     // retries on ANY error, including timeout
8      await sleep(backoff(i))
9    }
10  }
11  throw lastErr
12}