Interview prompt
Problem context
Skills being evaluated
Use the sequence below to surface constraints, choose boundaries, test failure behavior, and defend trade-offs. Concrete numbers are interview assumptions, not claims about a real production system.
Clarify the decision
- Start with business journeys and define which outcomes must survive: checkout may outrank recommendations, but fraud checks may still be mandatory. Establish legal, safety, and data-correctness functions that cannot be bypassed even in brownout.
Establish scale assumptions
- Model fan-out amplification, concurrency, pool occupancy, retry budgets, and queueing delay across dependency levels. A two-times arrival spike can create a twenty-times internal call spike when retries and scatter-gather paths multiply.
Functional and non-functional requirements
- Protect critical journeys, reject work before resources are consumed, preserve control traffic, and recover without a thundering herd. Degradation decisions must be centrally understandable but locally executable.
High-level architecture
- Define a versioned priority taxonomy and propagate deadline, priority, tenant, and cost budget in request context. Edge admission enforces global capacity; each service reserves critical pools, sheds optional work, and consumes a bounded per-request retry budget.
Data model and flow
- A request carries an absolute deadline and remaining work budget through every hop. Services return typed overload signals, not ambiguous timeouts; clients choose a documented fallback, enqueue, or fail-fast response based on journey policy.
Consistency and transaction boundaries
- Brownout may omit derived or stale-tolerant features but cannot weaken monetary, authorization, or inventory invariants. Queueing a mutation requires durable acceptance and idempotency; otherwise the system must clearly reject it.
Failure modes and recovery
- Coordinate circuit thresholds with dependency capacity and use adaptive concurrency limits near the saturated resource. Recovery ramps admitted work gradually, drains priority lanes first, and keeps retries jittered and budgeted.
Security and privacy
- Priority is assigned by trusted policy, never accepted directly from customers. Emergency bypasses are narrow, time-bound, audited, and incapable of skipping mandatory security or compliance decisions.
Observability and SLOs
- Build journey-level SLOs plus load-shed, fallback-quality, deadline-exhaustion, retry-amplification, and saturation metrics. Correlate protections across the graph so operators see whether a local breaker preserved or damaged the global outcome.
Capacity and cost
- Keep reserved capacity for control and critical journeys, while allowing normal borrowing before an incident. Quantify the revenue and customer cost of each degradation level so overprovisioning and brownout policy are business decisions.
Alternatives and trade-offs
- Central admission sees global demand but not every local bottleneck; local adaptive control reacts quickly but can produce inconsistent user outcomes. Combine a shared policy vocabulary with local enforcement and typed feedback.
Evolution and migration
- Instrument deadlines and retry budgets first, then introduce typed overload responses, shadow admission, optional-feature brownouts, and finally automated ramps. Run dependency-failure exercises with executives agreeing to the product degradation order in advance.
What Staff and Principal candidates should emphasize
- Distinguished-level reasoning spans architecture, product policy, and organizational governance. The answer should show how 180 teams converge on shared semantics while retaining local control, and how protections are tested as one system.
Decision trade-offs
Control model
Option A
Centralized global admission only
Option B
Independent local circuit breakers only
Recommendation:Use layered control: global admission for total work and local adaptive limits for the actual saturated dependency, connected by typed overload signals.
Capacity protection
Option A
Permanent hard reservations
Option B
Borrowable priority pools
Recommendation:Make critical floors non-negotiable but allow bounded borrowing with fast revocation so normal utilization does not collapse.
Follow-up interview questions
- 01Who decides the business order in which features degrade?
- 02How do you prevent every service from retrying a typed overload response?
- 03What signals should trigger gradual recovery rather than immediate full admission?
- 04How do you test degradation paths that span dozens of independently owned services?
Common weak answers and mistakes
- 01Listing circuit breakers without a coherent journey or priority policy.
- 02Letting clients self-assert critical priority and defeating fairness.
- 03Allowing each hop a fresh retry budget and multiplying load.
- 04Treating recovery as simply closing breakers once error rate falls.
Interviewer evaluation rubric
Adds generic retries and breakers but cannot reason about amplification, priority, invariants, or coordinated recovery.
Defines critical journeys, admission control, bounded retries, bulkheads, and basic brownout responses.
Propagates deadlines and cost budgets, combines global and local control, protects invariants, and measures fallback quality.
Creates an organization-wide overload contract tied to business value, testable cross-service exercises, and stable recovery dynamics.