Contain cascading failures across services that share hidden dependencies

• Trace the critical journeys and inventory shared databases, pools, DNS, identity, queues, and deployment dependencies. Define which requests may degrade, queue, or fail and which invariants cannot be bypassed.

• Estimate fan-out, concurrent connections, timeout occupancy, retry amplification, and queue growth during an eight-minute slowdown. Model the p99 dependency latency that exhausts each upstream pool.

• Keep critical paths responsive, fail optional work quickly, preserve control traffic, and recover gradually. One dependency or workload class must not consume every thread, connection, or retry token.

• Add per-dependency bulkheads, end-to-end deadlines, bounded queues, adaptive concurrency, circuit breakers, and admission before expensive fan-out. Isolate background and interactive pools and provide documented fallbacks.

• Propagate deadline, priority, tenant, and retry budget through calls. Return typed overload or degraded responses so callers do not mistake intentional shedding for a transient error worth retrying.

• Fallbacks may serve stale or partial derived data but cannot skip authorization, money, or inventory invariants. Deferred writes require durable acceptance and idempotent replay.

• Open breakers on measured saturation, not every error, and probe recovery through a small half-open budget. Drain queued work by priority and ramp admission slowly to avoid a second collapse.

• Keep authentication and abuse controls in the protected critical pool. Do not let a degradation flag bypass security checks or allow callers to forge priority.

• Measure pool occupancy, deadline exhaustion, retry amplification, breaker state, queue age, shed rate, and journey SLOs. Correlate by dependency to reveal hidden shared failure domains.

• Reserve a small floor for critical and control traffic, with borrowable normal capacity. Compare isolation cost against the revenue and recovery cost of a common-pool outage.

• Aggressive fail-fast protects capacity but may reject recoverable work; deep queues preserve requests but convert overload into latency and memory pressure. Choose explicitly per workload class.

• Instrument and cap retries first, split pools next, then introduce shadow admission and tested fallbacks. Run a game day that slows the dependency rather than simply killing it.

• Strong candidates explain how latency becomes concurrency and why recovery is a controlled phase. They identify hidden shared resources instead of treating every service box as isolated.

Interviewer evaluation rubric