Design a Notification System
Deliver push, SMS, and email at scale with fan-out, rate limits, retries, and dedup.
Overview
A notification system delivers messages across channels (push, SMS, email, in-app) triggered by events. It must fan out to the right recipients, respect user preferences and rate limits, handle third-party provider failures with retries, and never spam.
Requirements
Functional
- Send push (APNs/FCM), SMS, email, and in-app notifications.
- Respect per-user channel preferences and opt-outs.
- Support templated content and localization.
- Deduplicate and rate-limit to avoid spamming a user.
Non-functional
- High throughput with async delivery — callers shouldn't block on providers.
- At-least-once delivery with dedup; retries with backoff for provider failures.
- Observability: track sent/delivered/failed per channel.
Back-of-the-envelope
The numbers that justify the architecture.
| Notifications / day | 1B | |
| Peak send rate | ~50K/sec | |
| Channels | 4 | push, SMS, email, in-app |
| Provider timeout budget | few seconds w/ retry |
Reference architecture
Producers publish events; a notification service applies preferences/templates, enqueues per-channel work, and channel workers call third-party providers with retry.
Deep dives
Why everything is async and queued
Third-party providers (APNs, Twilio, SES) are slow and occasionally down. Never call them inline from a request. The notification service validates, applies preferences and templates, then drops work onto per-channel queues. Channel workers consume at a controlled rate, isolating a slow SMS provider from push. This decoupling is the whole point — a provider outage backs up one queue, not your product.
Idempotency and dedup
Events can be delivered more than once and retries re-enqueue work. Attach an idempotency key (event id + user + channel) and record delivered keys in a fast store with a TTL; workers skip duplicates. This prevents the classic 'user got the same push 5 times' incident.
Retries, backoff, and DLQ
On provider failure, retry with exponential backoff and jitter. After N attempts, move the message to a dead-letter queue for inspection rather than looping forever. Distinguish retryable (5xx, timeout) from permanent (invalid token → unsubscribe the device) failures.
Preferences, rate limits, and quiet hours
Before enqueuing, check user preferences (opted-out channels), frequency caps (max N/day), and quiet hours by timezone. A collapsing/digest strategy batches low-priority notifications so users aren't overwhelmed.
Key trade-offs
For each decision: the two options, and when to pick which.
Bottlenecks & follow-ups
Where it breaks under load — and what an interviewer will probe.
- ▲Slow provider blocks others → isolate per-channel queues and worker pools.
- ▲Duplicate sends → idempotency keys with TTL.
- ▲Retry storms → exponential backoff + jitter + DLQ.
What a strong answer sounds like
- ✓Justify queues immediately with the 'providers are slow and flaky' argument.
- ✓Bring up idempotency unprompted — duplicate notifications are the classic failure.
- ✓Mention preferences/rate limits; product sense scores points here.