Eliminate a retry storm during a long partial outage

• Classify failures as terminal, throttled, unknown outcome, and transient; decide which layer owns the retry for each operation. Define customer deadlines and provider idempotency support.

• Calculate total attempts from every layer and model backlog release at recovery. A three-attempt policy at four layers can create eighty-one calls for one logical request.

• Bound attempts per logical operation, preserve idempotency, honor provider Retry-After, stop work past the customer deadline, and recover at a controlled rate.

• Propagate one retry budget and absolute deadline, assign a single retry owner, use exponential backoff with full jitter, adaptive concurrency, and a circuit breaker. Move delay-tolerant operations to a durable scheduled queue.

• Carry operation and attempt IDs across layers; persist unknown payment outcomes and reconcile with the provider before another mutation. Typed errors tell callers whether to retry, poll, compensate, or stop.

• Retries are safe only with a stable idempotency key scoped to the business operation and retained through the provider’s ambiguity window. A timeout is not proof of failure.

• When the breaker opens, reject or durably defer rather than spinning. On recovery, release a token-limited fraction of backlog, prioritize fresh contractual work, and expire requests whose value has lapsed.

• Do not expose provider internals or accept client-controlled attempt counts. Sign queued commands and protect idempotency records from cross-tenant collisions.

• Measure attempts per operation, retry reasons, budget exhaustion, unknown outcomes, breaker state, backlog age, and recovery re-saturation. Separate original demand from retry traffic.

• Durable deferral costs storage but prevents compute and provider fees from multiplying. Size idempotency retention and delayed queues to the longest plausible outage plus investigation window.

• Hedged requests reduce tail latency for safe reads but amplify load and are dangerous for mutations. Retries improve transient success only while spare capacity exists; under saturation they must yield to admission control.

• Publish a shared retry contract, remove nested retries layer by layer, and canary with attempt telemetry. Fault-inject mixed responses and a slow recovery ramp, not only a clean outage.

• The key is one logical-operation budget across the call graph and explicit unknown-outcome handling. Staff answers also govern clients and workers, not just one service library.

Interviewer evaluation rubric