Apache Kafka
When to Use Kafka
Recognise the workloads Kafka is built for — and the ones it is wrong for.
Kafka shines as a high-throughput, durable event-streaming platform where data is read by many independent consumers, where replay of history matters, and where you want to retain events as a source of truth. Throughput in the millions of messages per second, horizontal scale via partitions, and long retention are its signature strengths.
Kafka is a freight rail network — enormous sustained throughput along fixed lines, unbeatable at volume, but overkill and clumsy for delivering a single parcel across town, where a courier (a queue) is the right tool.
Key Concepts
1
It is a poor fit when you need classic broker features: complex routing, per-message priority, per-message TTL, or per-message acknowledge/redelivery with delay. It is heavier to operate than a managed queue, and for simple point-to-point work with modest volume a queue like SQS or RabbitMQ is simpler and cheaper.
2
The decision usually comes down to: do you have a stream consumed many ways with replay needs (Kafka), or discrete tasks/commands handed to workers (a queue)?
3
What the interviewer is probing.
What the interviewer is probing.
4
1. "When is Kafka the wrong choice?" *Probing:* whether you can argue against it. *Stalls:*
"Never." *Moves up:* low volume, simple task queues, or when you need per-message acknowledgement
and priority — a queue broker is simpler and cheaper.
1. "When is Kafka the wrong choice?"
5
2. "What does Kafka give you that a queue does not?" *Probing:* the strengths. *Stalls:*
"Speed." *Moves up:* replayable retained history, many independent consumer groups over the same
data, and very high throughput.
2. "What does Kafka give you that a queue does not?"
6
3. "What is the operational cost?" *Probing:* honesty. *Stalls:* "It is managed." *Moves up:*
partition and retention planning, consumer lag monitoring, and rebalance tuning — more than a
managed queue needs.
3. "What is the operational cost?"
7
4. "How would you justify it to a sceptical architect?" *Probing:* the communication. *Stalls:*
"It is standard." *Moves up:* by the access pattern — several consumers needing the same stream and
the ability to replay is the case a queue cannot serve.
4. "How would you justify it to a sceptical architect?"