Back to System design
Message Queues: Kafka vs RabbitMQ vs SQS
mediumA message queue lets one part of your system hand work to another without waiting for it. The sender drops a message and moves on; a receiver picks it up later.
ScaleKafka: 1M+ msg/s/cluster; RabbitMQ: ~50K msg/s/node; SQS: 100K/s standard
StorageKafka: TB-scale on disk with retention; queues: drained continuously
Key Concepts
1
These three are compared constantly and they are not the same kind of thing.
2
Kafka is a log you read by position. Messages are appended to a file and stay there for days or weeks.
Reading does not remove anything — each reader keeps a bookmark saying how far it has got.
Kafka is a log you read by position.
3
That one property is why Kafka suits pipelines. A new team can start reading from the beginning. A broken
program can be fixed and rewound. Several teams can read the same messages independently without affecting
each other.
4
Order is guaranteed only within a partition, not across the whole topic. Kafka splits a topic into
partitions, and which partition a message goes to is decided by a key you choose. Key by user ID, and one
user's messages stay in order while different users spread across partitions.
5
RabbitMQ is a router that forgets. It sends each message to one or more queues based on rules you
configure, and once a consumer confirms it, the message is gone.
RabbitMQ is a router that forgets.
6
It does things Kafka does not: per-message priority, delays, and rich routing rules. It suits task queues —
"resize this image", "send this email". Throughput is lower than Kafka's because it tracks every message
individually, and a queue that grows very deep slows it down.
7
SQS is a queue you do not run. No servers, no partitions, no capacity planning. In exchange you accept
its behaviour: the standard version does not preserve order and may deliver a message twice; the FIFO
version preserves order but handles far less traffic.
SQS is a queue you do not run.
8
Messages are capped at 256 KB and kept for at most 14 days. When a consumer takes a message it becomes
invisible for a set time, and reappears if the consumer does not delete it — that timeout is how you avoid
two consumers doing the same work.
9
How many consumers can work at once? Kafka assigns partitions to consumers, so you cannot have more
working consumers than partitions. An extra one sits idle. Choose the partition count in advance.
How many consumers can work at once?
10
RabbitMQ and SQS hand individual messages to whoever is free, so you can add consumers freely. The trade is
that order is lost across them.
11
"Exactly once" is not a thing you can buy. Kafka can guarantee it within Kafka. The moment your
consumer writes to a database or calls a payment API, you are back to "at least once, possibly twice".
"Exactly once" is not a thing you can buy.
12
The real answer is to make the consumer safe to repeat: record the message ID with the work, and skip
anything already recorded.
13
Messages that always fail need somewhere to go. A message that crashes your consumer will be retried
forever unless you stop it. Retry a few times, then move it to a separate queue for failures, with the
reason attached. And read that queue — a failure queue nobody looks at is silent data loss.
Messages that always fail need somewhere to go.
14
What the interviewer is probing.
What the interviewer is probing.
15
1. "Which one would you use?" Probing: choosing on properties. Stalls: "Kafka, it scales." *Moves
up:* does anything need to re-read old messages, does order matter and at what level, do you want to run
servers.
1. "Which one would you use?"
16
2. "You need to reprocess yesterday's messages." Probing: whether reading deletes. Stalls:
"Re-send them." Moves up: Kafka rewinds a bookmark; the other two cannot, because the messages are gone.
2. "You need to reprocess yesterday's messages."
17
3. "How do you guarantee exactly once?" Probing: honesty. Stalls: "Turn on exactly-once." *Moves
up:* it holds inside one system; across an external call you need a consumer that is safe to repeat.
3. "How do you guarantee exactly once?"
18
4. "Your consumer is falling behind." Probing: how to scale it. Stalls: "Add consumers." Past the
partition count, Kafka ignores you. Moves up: plan partitions ahead, batch the work, and shed or
dead-letter poison messages.
4. "Your consumer is falling behind."
19
5. "One message keeps failing." Probing: poison messages. Stalls: "It retries." Moves up: a few
retries, then a dead-letter queue with the reason, and an alert so somebody reads it.
5. "One message keeps failing."
Components
- Producers with idempotency keys, batching, compression.
- Broker cluster with replicated partitions (Kafka: ISR-based replication).
- Consumer groups with rebalance protocol.
- Schema registry (Avro, Protobuf).
- DLQ + retry topic for poison messages.
- Lag monitoring + autoscaling consumers.
- Connectors — sinks to S3, Elasticsearch, JDBC; sources from CDC.
Properties compared
Kafka RabbitMQ SQS
-------------------------------------------------------------------
Model partitioned log broker + queues managed queue
On consume offset moves; message deleted deleted on
data remains delete call
Replay yes, to any no no
retained offset
Ordering per partition per queue none (Standard)
per group (FIFO)
Throughput very high moderate high (Standard)
(100s MB/s per (10k-50k msg/s low (FIFO,
broker) per queue) 3k msg/s)
Max message ~1 MB default large (not 256 KB
advised)
Retention time/size based until consumed 14 days max
(hours to weeks)
Routing topic + key rich (direct, none
topic, fanout,
headers)
Ops burden high (brokers, medium none
partitions, ZK/
KRaft)
Best at streams, replay, task queues, decoupling on
many independent complex routing, AWS with zero
consumers RPC operationsPartitions and parallelism
Kafka: parallelism is bounded by partitions
Partitions 12
Consumers in the group 12 -> 1 each
Add a 13th consumer -> sits idle
-> partition count is a capacity decision made in advance;
increasing it later changes key-to-partition mapping and
breaks ordering guarantees for in-flight keys. Sizing partitions
Target throughput 600 MB/s
Per-partition practical throughput ~10 MB/s
Minimum partitions 600 / 10 = 60
Add headroom for growth and skew = ~90-120 Cost of too many
Each partition is files, memory and a leader election unit.
Tens of thousands across a cluster slows recovery and
rebalancing measurably. RabbitMQ / SQS: competing consumers
Any consumer takes the next available message, so adding
consumers adds throughput without planning -- and ordering is
lost across them. FIFO groups in SQS restore ordering per group
at a much lower rate limit. The rebalance stall (Kafka)
When a consumer joins or leaves, partitions are reassigned and
consumption pauses briefly. Frequent restarts or long GC pauses
turn this into visible lag; cooperative rebalancing reduces it.Delivery semantics in practice
At-most-once
Acknowledge before processing. A crash loses the message.
Acceptable for: metrics samples, presence pings. At-least-once (the realistic default)
Acknowledge after processing. A crash before the ack means
redelivery -- so the consumer MUST be idempotent.
Implementation: a processed-message table keyed by message id,
written in the same transaction as the side effect. Exactly-once, honestly
Kafka transactions give it for read-process-write entirely
within Kafka (consume, transform, produce, commit offsets
atomically).
Crossing the boundary -- writing to Postgres, charging a card,
sending an email -- puts you back to at-least-once, because two
systems cannot commit atomically without 2PC.
So: effectively-once = at-least-once delivery + idempotent
consumer. Say it that way. Visibility timeout (SQS), the equivalent trap
Message hidden for N seconds while a consumer works. If the work
takes longer than N, the message reappears and a SECOND consumer
starts it. Fix: extend the timeout while working (heartbeat), or
set it above the p99 processing time -- and stay idempotent.