Back to System design

Message Queues: Kafka vs RabbitMQ vs SQS

medium
Scale: Kafka: 1M+ msg/s/cluster; RabbitMQ: ~50K msg/s/node; SQS: 100K/s standard Storage: Kafka: TB-scale on disk with retention; queues: drained continuously LinkedIn, Uber, Netflix
FundamentalsMessagingAsync

Message queues decouple producers and consumers so work can be done asynchronously, retried on failure, distributed across workers, and absorbed in bursts. The category splits into two architectures with very different properties.

ScaleKafka: 1M+ msg/s/cluster; RabbitMQ: ~50K msg/s/node; SQS: 100K/s standard
StorageKafka: TB-scale on disk with retention; queues: drained continuously

Key Concepts

1
1. Log-based vs queue-based. Log-based (Kafka, Pulsar, Kinesis, RedPanda): messages are immutable entries in a partitioned, durable log. Consumers track offsets and can replay. Multiple consumer groups read independently. Throughput scales with partitions. Queue-based (RabbitMQ, SQS, ActiveMQ): messages are work items pushed to a worker, ack'd, and deleted. Richer routing (exchanges, topics) but no replay.
1. Log-based vs queue-based.
2
2. Delivery semantics. At-most-once: fire and forget; messages can be lost (rare in practice). At-least-once: producer retries until ack; consumers may see duplicates. The practical default — pair with idempotent consumers. Exactly-once: achievable in Kafka with idempotent producers + transactions, but expensive. Most production systems use at-least-once + idempotency keys at the consumer.
2. Delivery semantics.
3
3. Ordering is per-partition. Kafka guarantees order within a partition, not globally. Route related events to the same key (user_id, order_id) so they land on one partition. Global ordering = single partition = no parallelism = throughput ceiling of one node. Don't promise it.
3. Ordering is per-partition.user_idorder_id
4
4. Operational essentials. Batching on producers cuts broker round-trips. Backpressure via consumer lag (messages-behind-head) + autoscaling consumers. Dead-letter queue for poison messages — otherwise they wedge consumers permanently. Schema registry (Avro / Protobuf) with semver compatibility prevents producer changes from breaking consumers. Retention: Kafka commonly 7 days to weeks; SQS deletes on ack.
4. Operational essentials.
5
5. When to pick what. Event sourcing / analytics: Kafka (replay, multi-consumer). Task queues (image resize, email): RabbitMQ or SQS. Fanout to many services: Kafka topics or SNS+SQS. Fully managed, simplest ops: SQS. Production references: LinkedIn (built Kafka), Uber, Netflix, Confluent.
5. When to pick what.

Related Problems

CAP Theorem and PACELCeasyLoad Balancing: L4 vs L7, Algorithms, Health CheckseasyCaching Strategies: Cache-aside, Write-through, Write-backeasySQL vs NoSQL: When to Pick WhicheasySharding Strategies: Range, Hash, Directory, Geomedium

Approach

  1. Pick the family. Log-based (Kafka, Pulsar) for events, replay, multi-consumer. Queue (RabbitMQ, SQS) for task distribution.
  2. Partition for throughput. Kafka: hash by routing key. SQS FIFO: MessageGroupId. RabbitMQ: multiple queues with consistent-hash exchange.
  3. Choose delivery semantics. Default at-least-once + idempotent consumers. Exactly-once only when truly needed.
  4. Plan for poison messages: DLQ with explicit replay tooling.
  5. Schema discipline: Avro/Protobuf + registry; semver compatibility rules.
  6. Producer config: batching, compression (Snappy/LZ4), acks=all for durability.
  7. Consumer config: explicit commit on success; commit interval tuned for at-least-once.
  8. Monitor lag: messages-behind-head per consumer group; alert when lag grows.

Family-by-family

Kafka: append-only partitioned log. Consumers track offsets. Pull-based. Retention configurable. Strong throughput. Operationally heavy (ZooKeeper or KRaft). Used at LinkedIn, Netflix, Uber.

Pulsar: like Kafka but with multi-tenancy, tiered storage, separated compute/storage. Used at Yahoo, Tencent.

RabbitMQ: queues with rich routing (exchanges, bindings). Push-based delivery, ack/nack semantics. Strong for task queues. Smaller scale than Kafka.

SQS: fully managed, no ordering (standard) or limited ordering (FIFO). No replay. Easy to operate. Use SNS+SQS for fanout.

NATS JetStream: newer, lightweight, good defaults. Strong for microservices messaging.

Components

  • Producers with idempotency keys, batching, compression.
  • Broker cluster with replicated partitions (Kafka: ISR-based replication).
  • Consumer groups with rebalance protocol.
  • Schema registry (Avro, Protobuf).
  • DLQ + retry topic for poison messages.
  • Lag monitoring + autoscaling consumers.
  • Connectors — sinks to S3, Elasticsearch, JDBC; sources from CDC.

Trade-offs

Kafka: highest throughput, replay, ordering per key. Operational complexity is real (ZK/KRaft, brokers, partitions, retention, compaction).

RabbitMQ: rich routing, simpler ops, smaller scale.

SQS: simplest, fully managed. No replay, limited ordering.

Exactly-once: achievable in Kafka but expensive. Idempotency at the consumer is the pragmatic alternative.

Global ordering = no parallelism. Don't promise it. Design ordering as a per-key property.

Retention vs cost: longer retention enables replay and audit; consumes disk.

Common pitfalls

  • Treating SQS like Kafka. SQS doesn't replay.
  • Using a single partition for ordering — that's a throughput ceiling of one node.
  • Forgetting DLQ — poison messages wedge consumers permanently.
  • Letting consumer lag grow indefinitely — eventually retention drops the unread messages.
  • No schema discipline — one bad producer change breaks every consumer.
  • Manual offset management — let the consumer library handle it unless you have a very good reason.