Interview prompt
Problem context
Skills being evaluated
Use the sequence below to surface constraints, choose boundaries, test failure behavior, and defend trade-offs. Concrete numbers are interview assumptions, not claims about a real production system.
Clarify the decision
- Define the business operation and scope before choosing a key. Ask whether repeated requests with changed payload should conflict, return the first result, or represent a new operation.
Establish scale assumptions
- Estimate operations, retry horizon, replay and callback delays, key cardinality, and stored outcome size. TTL must exceed the longest ambiguity and legal retry window, not an arbitrary cache duration.
Functional and non-functional requirements
- One logical payout executes once despite transport retries, concurrent attempts receive a stable result, payload mismatch is detected, and historical replay cannot bypass deduplication.
High-level architecture
- Create a canonical operation service or library contract that maps source identity to a scoped operation ID. Persist idempotency record, request fingerprint, state, and business side effect in the same transaction; publish through an outbox.
Data model and flow
- Adapters normalize HTTP keys, message IDs, callback references, and batch row identities into the canonical operation. Consumers pass the operation ID through every hop and return stored terminal or in-progress status.
Consistency and transaction boundaries
- Use a unique constraint and transactional state transition, not read-then-write. External effects need provider idempotency keys plus reconciliation because local atomicity cannot include the partner.
Failure modes and recovery
- A crash after effect but before response returns the stored outcome on retry. Stuck in-progress operations have leases or owner epochs and a reconciliation path, not blind re-execution.
Security and privacy
- Scope keys by tenant and operation type, prevent callers from probing another customer’s outcomes, and avoid storing sensitive request bodies as fingerprints. Authenticate callbacks before identity mapping.
Observability and SLOs
- Track dedup hits, payload conflicts, in-progress age, expired-key replays, provider ambiguity, and operations with multiple source identities. Alert on any terminal side effect lacking a canonical operation record.
Capacity and cost
- Store compact outcome references and tier old records, but retain financial operation identity for the required audit horizon. Bloom filters may optimize old lookups but never replace authoritative records.
Alternatives and trade-offs
- Centralizing all operations simplifies identity but adds latency and ownership concentration; a shared contract with domain-local stores scales better if scope and retention are enforced consistently.
Evolution and migration
- Backfill mappings for active operations, require canonical IDs on new producers, run conflict detection in shadow, then block unscoped keys. Replays use explicit original operation IDs and dry-run reports.
What Staff and Principal candidates should emphasize
- Principal candidates define idempotency as stable business outcome, not duplicate-message suppression. They cover concurrent requests, payload mismatch, TTL, external ambiguity, and replay governance.
Decision trade-offs
Dedup scope
Option A
Transport-specific message identifier
Option B
Canonical business-operation identifier
Recommendation:Use the business operation as authority; transport IDs are evidence mapped into it, because the same operation crosses transports.
Retention
Option A
Short cache-style TTL
Option B
Retention matched to ambiguity and business risk
Recommendation:Retain high-consequence operation identities for the full retry, replay, and audit window even if detailed payloads are tiered away.
Follow-up interview questions
- 01What should happen when the same key arrives with a different payload?
- 02How do you recover an operation stuck in progress?
- 03Can an outbox alone guarantee an external payout happens once?
- 04How do you replay a year of events safely?
Common weak answers and mistakes
- 01Using a queue message ID as the business deduplication identity.
- 02Checking for a key before a separate side-effect transaction.
- 03Expiring financial idempotency records after a convenient twenty-four hours.
- 04Treating an external timeout as proof the provider did nothing.
Interviewer evaluation rubric
Stores recent message IDs in a cache and cannot handle concurrent attempts, alternate transports, or external outcomes.
Defines a scoped operation key, transactional unique record, stored outcome, payload conflict, and provider reconciliation.
Normalizes identities across transports, handles leases and ambiguity, sets risk-based retention, and governs replay.
Creates a domain-aligned cross-team operation contract that survives migrations, audits, provider behavior, and years-later recovery.