Interview prompt
Problem context
A workflow platform processes billions of state transitions per day. Product asks that events be handled “in order,” but global serialization cannot meet throughput or regional latency goals. Some transitions are independent; others update the same account or workflow instance. Define ordering guarantees and execution architecture.
Skills being evaluated
ordering semanticspartition designstate machinesrecovery protocols
A reasoning guide, not a memorized architecture
Use the sequence below to surface constraints, choose boundaries, test failure behavior, and defend trade-offs. Concrete numbers are interview assumptions, not claims about a real production system.
01
Clarify the decision
- Ask what business error occurs when two events reorder and identify the smallest key that shares that invariant. Distinguish producer order, broker order, processing order, and visible state order; they are often incorrectly collapsed.
02
Establish scale assumptions
- Assume 2 billion events per day, ten-times bursts, and a power-law key distribution. Estimate throughput per ordering key and identify single hot workflows that cannot gain parallelism merely by adding partitions.
03
Functional and non-functional requirements
- Guarantee monotonic transitions per workflow or account, accept at-least-once transport, and provide a bounded way to recover gaps and poison events. Unrelated keys must progress independently.
04
High-level architecture
- Hash the ordering key to a partition, execute one leased state-machine owner per key range, and persist state with the consumed sequence in one conditional transaction. A gap buffer and repair service fetch missing events without blocking unrelated partitions.
05
Data model and flow
- Events carry operation ID, ordering key, producer epoch, sequence, schema version, and causation ID. The processor compares the expected sequence, applies a valid transition idempotently, and writes state plus checkpoint atomically before acknowledgment.
06
Consistency and transaction boundaries
- Ordering is scoped per key; cross-key workflows use sagas and explicit compensation rather than global order. Fencing epochs prevent a paused owner from committing after reassignment, while optimistic versions reject stale transitions.
07
Failure modes and recovery
- On duplicates, return the stored outcome; on gaps, quarantine only that key and trigger targeted replay. On partition reassignment, wait for or fence the prior lease, restore from checkpoint, and make side effects idempotent through an outbox.
08
Security and privacy
- Authorize transitions against current tenant and actor context, not just a previously trusted event. Protect sequence metadata from tenant manipulation and ensure repair tooling cannot replay another tenant’s stream.
09
Observability and SLOs
- Measure gap age, duplicate rate, fenced writes, per-key lag, hot-key skew, invalid transitions, and time in compensation. Define a freshness SLO per workflow class instead of one aggregate consumer-lag number.
10
Capacity and cost
- Partition for expected peak per key range with virtual shards that can move, while giving pathological keys a serial but isolated lane. Retain the event log only as long as replay and audit requirements need; compact state separately.
11
Alternatives and trade-offs
- More ordering scope simplifies some application logic but destroys parallelism and availability. Per-key ordering pushes cross-key coordination into the domain model, which is honest and scalable when compensation is explicit.
12
Evolution and migration
- First add operation IDs and state versions, then record checkpoints beside state, and only then repartition. Dual-consume the old and new partition maps in shadow and compare terminal state before cutover.
13
What Staff and Principal candidates should emphasize
- Principal-level answers challenge the word “order” and convert it into invariant-specific guarantees. They cover fencing, gaps, hot keys, atomic checkpointing, and the user-visible behavior of replay and compensation.
Decision trade-offs
Ordering scope
Option A
Global or tenant-wide total order
Option B
Per-entity ordering with sagas across entities
Recommendation:Choose the narrowest key that contains the invariant; global order is justified only for a truly global scarce resource.
Gap handling
Option A
Block the entire partition
Option B
Park the affected key and continue others
Recommendation:Park and repair per key when storage supports it; blocking a broad partition turns one missing event into a large availability incident.
Follow-up interview questions
- 01How do you handle one ordering key that alone exceeds a partition’s throughput?
- 02What fences a consumer that resumes after its lease has expired?
- 03Can two producer services safely allocate sequences for the same key?
- 04How do you prove a replay did not repeat an external side effect?
Common weak answers and mistakes
- 01Claiming Kafka or another broker provides end-to-end ordering automatically.
- 02Requesting global order without identifying the invariant it protects.
- 03Committing the checkpoint separately from state and losing atomic progress.
- 04Ignoring hot keys that remain serial regardless of partition count.
Interviewer evaluation rubric
Weak
Relies on broker partitions alone and does not distinguish delivery order from committed business state.
Solid
Defines a per-key order, idempotent processing, checkpoints, and a basic replay strategy.
Strong
Adds fencing epochs, atomic state transitions, gap isolation, hot-key handling, outbox effects, and scoped SLOs.
Exceptional
Derives ordering scope directly from invariants and provides a credible repartition and verification plan without global coordination.