Interview prompt
Problem context
Skills being evaluated
Use the sequence below to surface constraints, choose boundaries, test failure behavior, and defend trade-offs. Concrete numbers are interview assumptions, not claims about a real production system.
Clarify the decision
- Define RPO and RTO per business journey, not as one marketing number. Identify dependencies, data sets, and external contracts that cannot move and decide who has authority to declare evacuation.
Establish scale assumptions
- Calculate replication lag, DNS and client-cache delay, cold-start time, backlog, and passive capacity under peak primary load. The standby must absorb traffic plus replay and repair work.
Functional and non-functional requirements
- Fence the failed writer, preserve the stated loss bound, restore critical journeys first, communicate degraded features, and produce evidence from regular exercises. Return home is a separate migration with its own gates.
High-level architecture
- Maintain warm regional infrastructure, replicated configuration and secrets, tested data promotion, global traffic control, and a failover coordinator that records an ownership epoch. Catalog every external dependency and preapprove standby identities.
Data model and flow
- Continuously replicate durable state with lag telemetry and immutable checkpoints. On evacuation, stop or fence writes, promote from a known position, switch traffic by cohort, and reconcile ambiguous requests using operation IDs.
Consistency and transaction boundaries
- Choose asynchronous replication only if the one-minute data-loss risk is accepted; otherwise coordinate the critical write subset. The promotion epoch prevents the old region from accepting writes after network recovery.
Failure modes and recovery
- Prepare for partial evacuation, stale DNS, duplicated webhooks, and a primary that is isolated rather than dead. Keep enough capacity for rollback or controlled fail-forward and avoid automatic failback.
Security and privacy
- Preprovision certificates, keys, allowlists, and least-privilege identities in standby; break-glass actions expire and are audited. Residency rules may forbid a nominal failover target.
Observability and SLOs
- Measure real replication lag, promotion phase duration, traffic by region, error and correctness SLOs, backlog, and dependency readiness. Time every quarterly exercise against declared objectives.
Capacity and cost
- Warm standby is an insurance product; choose reserved critical capacity plus autoscaling for optional tiers. Track the gap between paid standby and actually exercisable throughput.
Alternatives and trade-offs
- Active-active reduces evacuation work but creates continuous consistency and cost complexity. Warm active-passive is often enough if promotion, capacity, and external dependencies are genuinely rehearsed.
Evolution and migration
- Start with dependency inventory and read-only traffic, then practice service cohorts, write promotion, and full regional loss. Fix each exercise finding before broadening the public RTO claim.
What Staff and Principal candidates should emphasize
- Staff candidates make RPO and RTO test results, not architecture adjectives. They cover fencing, ambiguous writes, third parties, capacity, communications, and the dangerous return-home phase.
Decision trade-offs
Standby posture
Option A
Cold or minimally provisioned standby
Option B
Warm capacity with continuous verification
Recommendation:Use warm capacity for a 15-minute RTO; a cold environment cannot credibly satisfy dependency, cache, and scale readiness.
Failback
Option A
Automatically return when primary health recovers
Option B
Treat return as a separately gated migration
Recommendation:Gate return after data reconciliation and stability; automatic failback turns a recovering incident into split brain or repeated oscillation.
Follow-up interview questions
- 01How do you fence a primary that cannot be reached but may still serve local clients?
- 02What capacity must remain unused in the passive region?
- 03How do you reconcile writes accepted just before traffic switched?
- 04What evidence would justify publishing a 15-minute RTO?
Common weak answers and mistakes
- 01Equating replicated data with a complete regional failover capability.
- 02Ignoring third-party allowlists, certificates, and operational access.
- 03Promoting without fencing the former writer.
- 04Automatically failing back before reconciliation and incident stability.
Interviewer evaluation rubric
Points DNS at a replica but cannot meet data, capacity, dependency, or authority-transfer requirements.
Defines replication, promotion, traffic shift, warm capacity, runbooks, and regular failover exercises.
Adds per-journey objectives, fencing epochs, ambiguous-write reconciliation, dependency readiness, and gated return.
Treats resilience claims as measured governance, narrows synchronous cost to critical state, and aligns product degradation with evidence.