Interview prompt
Problem context
Skills being evaluated
Use the sequence below to surface constraints, choose boundaries, test failure behavior, and defend trade-offs. Concrete numbers are interview assumptions, not claims about a real production system.
Clarify the decision
- Define ordering scope, retention and replay windows, maximum payload sensitivity, and whether events represent facts or commands. Ask what acknowledgment means and which customer failures are terminal.
Establish scale assumptions
- Assume high fan-out, a power-law endpoint distribution, and multi-hour customer outages. Size durable queues, retry storage, connection pools, and replay capacity independently from live traffic.
Functional and non-functional requirements
- Provide durable at-least-once delivery, stable event IDs, per-endpoint isolation, verifiable signatures, bounded retries, replay, and transparent delivery history. Never let one customer starve others.
High-level architecture
- Persist events and endpoint deliveries separately, partition by endpoint or ordering key, schedule through tenant-fair queues, and use isolated delivery workers with circuit breakers. A customer console and API expose attempts and replay.
Data model and flow
- Each delivery carries event ID, type, creation time, schema version, sequence where promised, and signature timestamp. Store attempt outcome before scheduling the next retry and deduplicate replay requests.
Consistency and transaction boundaries
- At-least-once is the honest guarantee; customers deduplicate by event ID. If order matters, serialize per documented key while allowing independent keys to progress.
Failure modes and recovery
- Honor Retry-After, classify 4xx terminal behavior, jitter exponential retries, cap age and attempts, and move exhausted deliveries to a visible failed state. Recovery admission prevents a customer backlog from overwhelming live events.
Security and privacy
- Sign raw bytes with timestamped rotating secrets, accept overlap during rotation, block SSRF destinations, and redact payloads in diagnostics. Replay protection belongs to the receiver’s timestamp and event-ID policy.
Observability and SLOs
- Expose delivery latency, attempts, response codes, endpoint health, backlog age, and dropped or expired events by customer. Internally track provider saturation and fairness.
Capacity and cost
- Charge or limit extreme retention and replay, compress stored payloads, and keep payload versus delivery metadata on different retention tiers. Egress and connection churn are major cost units.
Alternatives and trade-offs
- Strict per-endpoint order makes one poison event block progress; unordered delivery improves throughput but shifts causality to consumers. Offer ordering only for domains that need and can pay for it.
Evolution and migration
- Add event IDs and diagnostics before changing retries, then migrate signatures with dual verification, and introduce versioned event envelopes. Provide a test endpoint and contract fixtures for customers.
What Staff and Principal candidates should emphasize
- Staff candidates make failure visible to customers and refuse exactly-once claims. They cover SSRF, key rotation, backlog fairness, replay, and poison-event policy.
Decision trade-offs
Ordering
Option A
Strict order for every endpoint
Option B
Independent delivery with optional ordering keys
Recommendation:Use optional per-key order; global endpoint order creates head-of-line blocking for unrelated facts.
Payload storage
Option A
Retain complete payload for every attempt
Option B
Store one encrypted event payload plus attempt metadata
Recommendation:Store one protected payload and reference it from attempts, with retention matched to replay and privacy obligations.
Follow-up interview questions
- 01How does secret rotation avoid breaking in-flight deliveries?
- 02What should happen after a customer returns 400 for one event?
- 03How do live events compete with a week-old replay?
- 04Can you provide exactly-once effects to the customer?
Common weak answers and mistakes
- 01Promising exactly-once delivery over an unreliable network.
- 02Retrying all 4xx responses indefinitely.
- 03Signing reserialized JSON rather than the exact transmitted bytes.
- 04Using one global queue and allowing a large customer outage to dominate workers.
Interviewer evaluation rubric
Adds a queue and retries but omits deduplication, customer diagnostics, security, and per-endpoint isolation.
Defines event IDs, at-least-once delivery, jittered retries, signatures, DLQ behavior, and replay.
Adds fair scheduling, ordering domains, SSRF controls, key overlap, backlog recovery, and observable customer history.
Treats webhook delivery as a durable product contract with explicit cost, privacy, lifecycle, and receiver-operability design.