The verdict
Choose Kafka for high-throughput event streaming where you need replay and multiple independent consumers of the same stream. Choose RabbitMQ for flexible routing and classic task queues with per-message acknowledgment. Choose SQS when you want a zero-ops managed queue on AWS and don't need ordering or replay. The key insight: Kafka is a durable log, not a queue — messages aren't deleted when read — and that single difference drives most of the decision.
How to decide
- 1Do you need streaming, event sourcing, or replay, or is this a task queue? Log vs queue is the first fork.
- 2What throughput and retention do you need? Very high + retained → Kafka.
- 3What ordering guarantees does the workload actually require? Per-key ordering → Kafka partitions or SQS FIFO.
- 4What's your ops appetite? Want none and you're on AWS → SQS.
- 5How complex is your routing (fanout, topic-based)? Complex → RabbitMQ.
- 6What delivery semantics do you need — at-least-once (default nearly everywhere) or effectively-once? Design idempotent consumers regardless.
The options
Apache Kafka
A distributed, durable, replayable commit log.
Best for
- Event streaming and event sourcing
- Many independent consumers reading the same stream
- High throughput with retention and replay
Strengths
- Very high throughput; horizontal scale via partitions
- Messages are retained and replayable — consumers track their own offset
- Multiple consumer groups read the same topic independently
Weaknesses
- Operationally heavy to self-run (though managed options exist)
- Ordering is only per-partition; global ordering needs one partition
- Overkill for a simple work queue
RabbitMQ
A mature message broker with rich routing.
Best for
- Traditional task/work queues
- Complex routing (topic, fanout, header exchanges)
- Per-message acknowledgment and priority
Strengths
- Flexible routing via exchanges and bindings
- Per-message ack, redelivery, priorities, TTLs
- Lower conceptual overhead than Kafka for queue workloads
Weaknesses
- Lower peak throughput than Kafka
- Messages are typically consumed and removed — no built-in long-term replay
- You operate the cluster (unless managed)
Amazon SQS
A fully-managed, zero-ops queue on AWS.
Best for
- AWS-native services that need a simple, reliable queue
- Teams that want no infrastructure to run
- Decoupling producers and consumers with minimal effort
Strengths
- Fully managed — no brokers to run or scale
- Cheap, reliable, integrates with the AWS ecosystem
- FIFO queues offer ordering + exactly-once processing when needed
Weaknesses
- No message replay or long retention (max 14 days)
- Limited routing; standard queues are at-least-once and unordered
- AWS-only; FIFO has throughput limits
Trade-offs at a glance
| Dimension | Apache Kafka | RabbitMQ | Amazon SQS |
|---|---|---|---|
| Model | Durable log | Broker / queue | Managed queue |
| Throughput | Very high | High | High (FIFO limited) |
| Ordering | Per-partition | Per-queue | FIFO queues only |
| Replay / retention | Yes (configurable) | No (consumed) | No (≤14 days) |
| Routing | Topics + partitions | Rich exchanges | Basic |
| Consumer model | Consumer groups pull | Push/pull, ack | Poll, visibility timeout |
| Ops burden | High (or managed) | Medium | None (managed) |
| Best for | Event streaming + replay | Flexible task queues | Zero-ops AWS queue |
In the interview
In the asynchronous-processing or decoupling part of a design — 'how do these services communicate without blocking?' The interviewer wants to see you distinguish a log from a queue and pick on requirements.
What a Staff answer includes
A Staff answer separates the durable-log model (Kafka) from the queue model (RabbitMQ/SQS), reasons explicitly about delivery semantics, ordering, and replay, and designs idempotent consumers with a dead-letter path. It chooses on requirements and ops reality, not on what it has used before.
Follow-ups you should expect
- →How do you achieve exactly-once (or effectively-once) processing?
- →How do you handle a poison message that keeps failing?
- →What's your partition/ordering key, and how do you avoid a hot partition?
- →How do you scale consumers, and what happens when one falls behind?
Common mistakes
- ✕Reaching for Kafka for a simple task queue because it 'scales'.
- ✕Assuming exactly-once delivery for free instead of designing idempotent consumers.
- ✕Ignoring dead-letter queues and retry/backoff for failures.
- ✕Choosing a queue (no replay) when the requirement is really event sourcing.
Further reading
Related decisions
Synchronous vs Asynchronous Communication
Synchronous when the caller needs an immediate answer to proceed; asynchronous messaging to decouple services, absorb load, tolerate downstream failure, or fan out — at the cost of eventual consistency and more moving parts.
Strong vs Eventual Consistency
Strong consistency when correctness needs every reader to see the latest write (money, inventory, uniqueness); eventual when availability and scale matter more and brief staleness is tolerable (feeds, counts, caches).