Interview prompt
Problem context
Skills being evaluated
Use the sequence below to surface constraints, choose boundaries, test failure behavior, and defend trade-offs. Concrete numbers are interview assumptions, not claims about a real production system.
Clarify the decision
- List the current invariants and ask which must be immediate versus eventually repaired. Define reservation expiry, authorization versus capture, cancellation policy, and the customer-visible states during a delayed workflow.
Establish scale assumptions
- Assume 5k orders/second at peak, payment latency in hundreds of milliseconds, and occasional partner outages. Size reservation and retry storage for hours of backlog, not only steady-state throughput.
Functional and non-functional requirements
- Never capture payment without a durable order, prevent oversell within the promised policy, make every command idempotent, and expose a finite state machine to customers and support. Recovery must not depend on a human reading logs.
High-level architecture
- Use an order-owned saga: create pending order, reserve inventory, authorize payment, confirm order, then capture according to policy. Each service commits its local state and outbox event atomically; the orchestrator records durable step state and compensation.
Data model and flow
- Commands carry order and operation IDs, expected state, and deadlines. Events include causal IDs and versions; handlers deduplicate, reject invalid transitions, and emit outcomes rather than assuming delivery equals success.
Consistency and transaction boundaries
- Inventory reservation is strongly consistent within its partition, payment state is authoritative at the provider, and the cross-service outcome is eventually consistent. Compensation is a first-class business action, not database rollback.
Failure modes and recovery
- Handle timeout as unknown, query provider state before retrying capture, expire orphan reservations, and reconcile order, inventory, and payment ledgers. Poison workflows enter a visible manual-review state with safe operator commands.
Security and privacy
- Tokenize payment details, minimize sensitive data in events, authenticate service commands, and enforce tenant and order ownership at each service. Compensation actions need the same authorization and audit controls as forward actions.
Observability and SLOs
- Track time in each saga state, unknown payment outcomes, compensation rate, reservation expiry, reconciliation drift, and terminal success SLO. Trace by order and operation ID across services.
Capacity and cost
- Event and saga state add durable writes, but make correctness inspectable. Partition orchestration by order ID, bound retries, and retain detailed transitions according to financial audit needs.
Alternatives and trade-offs
- Two-phase commit preserves atomic appearance but couples availability and operational domains. A saga admits intermediate states and requires compensation, yet matches external payment reality and independent service ownership.
Evolution and migration
- Extract one responsibility while the original database remains authoritative, introduce outboxes and idempotency, then move commands behind an adapter. Shadow the new state machine and reconcile before allowing it to own customer-visible outcomes.
What Staff and Principal candidates should emphasize
- Staff candidates name the invariant owner, legal intermediate states, unknown outcomes, and support workflow. They do not say “eventual consistency” as if it automatically repairs money or inventory.
Decision trade-offs
Coordination
Option A
Distributed two-phase commit
Option B
Durable saga with compensation
Recommendation:Use a saga across independent services and external payments; reserve two-phase commit for tightly controlled stores with compatible failure domains.
Payment timing
Option A
Capture before inventory confirmation
Option B
Authorize then capture after confirmation
Recommendation:Authorize first and capture after durable confirmation when provider semantics allow, reducing refund and customer-trust failures.
Follow-up interview questions
- 01How do you handle a payment timeout where the provider may have captured funds?
- 02What prevents a late inventory confirmation after the reservation has expired?
- 03Who owns the terminal truth shown to customer support?
- 04How would you migrate existing in-flight orders into the new state machine?
Common weak answers and mistakes
- 01Treating compensation as guaranteed rollback rather than another fallible business operation.
- 02Retrying payment capture blindly after a timeout with unknown outcome.
- 03Publishing an event outside the local state transaction.
- 04Leaving intermediate states undocumented and invisible to customers or support.
Interviewer evaluation rubric
Draws three services and a queue but leaves partial success, unknown payment outcomes, and invariant ownership unresolved.
Defines a saga, local transactions, outboxes, idempotency, reservation expiry, and basic compensation.
Models a durable state machine, causal events, provider reconciliation, support states, security, and migration of in-flight work.
Challenges whether the split is worth its correctness cost and aligns service boundaries, team ownership, and business compensation policy.