Interview prompt
Problem context
Skills being evaluated
Use the sequence below to surface constraints, choose boundaries, test failure behavior, and defend trade-offs. Concrete numbers are interview assumptions, not claims about a real production system.
Clarify the decision
- Inventory operations by invariant rather than declaring the whole product strongly or eventually consistent. Ask which writes may conflict, which require read-your-writes, which identities must be globally unique, and whether degraded writes are preferable to rejection during partition.
Establish scale assumptions
- Assume three regions, 120k peak writes/second, 99.99% availability, and inter-region latency between 80 and 220 milliseconds as explicit interview inputs. Model tenant locality and the fraction of truly cross-region collaboration before paying coordination latency everywhere.
Functional and non-functional requirements
- Serve local reads and most writes, maintain residency, expose deterministic conflict behavior, and meet an agreed regional RPO and RTO. Global invariants must remain true even if that means a narrower unavailable surface during partition.
High-level architecture
- Use tenant home regions for regulated data, entity-affinity routing for collaborative objects, and multi-leader replication only for operations with a defined merge algebra. Keep a small globally coordinated service for scarce invariants such as unique namespace claims.
Regional data planes accept locally safe work while a narrow global coordination plane owns non-mergeable invariants.
Data model and flow
- Attach entity version, origin region, logical time, actor, and operation ID to replicated mutations. Consumers apply mutations idempotently; a reconciliation stream detects divergent versions and routes non-mergeable cases to a compensating workflow.
Consistency and transaction boundaries
- Choose semantics per operation: CRDT-style merge for commutative sets and counters, last-writer-wins only where lost updates are acceptable, and quorum or single-owner writes for hard invariants. Session tokens provide read-your-writes when users move between regions.
Failure modes and recovery
- During partition, keep owner-region writes and mergeable local operations available, but reject global uniqueness claims that cannot reach quorum. Regional evacuation requires fencing the old writer before promotion to prevent two legitimate primaries.
Security and privacy
- Residency policy participates in routing and replication authorization, not just storage location. Encrypt inter-region logs, restrict repair tooling by tenant and purpose, and record every conflict override as an auditable administrative action.
Observability and SLOs
- Track replication lag by tenant and stream, conflict rates by operation, rejected invariant writes, session-staleness violations, and evacuation time. Set separate SLOs for local availability and cross-region convergence.
Capacity and cost
- Budget for duplicate storage, inter-region egress, repair compute, and standby capacity during evacuation. Keep traffic local by default and replicate the minimum data needed for product semantics rather than every table to every region.
Alternatives and trade-offs
- Universal multi-leader storage maximizes write locality but forces every domain to solve conflicts. Home-region ownership is simpler and often sufficient; reserve global coordination for the small invariant surface that justifies its latency.
Evolution and migration
- Start with tenant home regions and async read replicas, instrument cross-region access, then enable active-active only for operations with tested merge rules. Backfill origin and version metadata before changing write routing, and rehearse fencing in production-like game days.
What Staff and Principal candidates should emphasize
- The senior signal is refusing a single consistency slogan. A Staff candidate maps business invariants to distinct coordination costs and can explain exactly what the user sees during a partition, failover, conflict, and repair.
Decision trade-offs
Write topology
Option A
Tenant or entity home-region ownership
Option B
Multi-leader writes in every region
Recommendation:Use ownership for non-commutative state and multi-leader only where merge semantics are explicit, testable, and visible to users.
Global invariants
Option A
Synchronous quorum coordination
Option B
Optimistic local claims with later repair
Recommendation:Coordinate hard uniqueness and monetary limits; use optimistic claims only when compensation is acceptable and product policy defines the loser experience.
Follow-up interview questions
- 01How do users retain read-your-writes after their traffic fails to another region?
- 02What fences a recovered region that still believes it owns writes?
- 03Which product operations are good CRDT candidates and which are not?
- 04How would you meet residency requirements while allowing a global support team to diagnose incidents?
Common weak answers and mistakes
- 01Calling a database active-active without defining conflict behavior for each write.
- 02Using last-writer-wins for invariants where losing a valid write is unacceptable.
- 03Discussing failover without a fencing token or authoritative ownership epoch.
- 04Replicating regulated data globally and treating residency as a legal footnote.
Interviewer evaluation rubric
Promises local writes and strong consistency everywhere without acknowledging network latency, partitions, or conflict semantics.
Chooses a sensible regional topology, distinguishes strong invariants from mergeable data, and defines basic failover.
Uses operation-specific semantics, fencing, session guarantees, residency-aware routing, and measurable reconciliation.
Connects consistency costs to product behavior, narrows the coordinated surface, and provides an incremental rollout with game-day evidence.