Apache Kafka
Architecture Overview
Understand Kafka as a distributed, partitioned, replicated commit log — not a traditional queue.
Kafka is best understood as a distributed append-only log rather than a message queue. Producers append records to the end of a log; consumers read forward at their own pace by tracking an offset. Because messages are not deleted when read, many independent consumers can replay the same data, and a slow consumer never blocks a fast one.
A newspaper archive: every edition is filed in order and never thrown away. Any number of readers can start from any past date and read forward at their own speed — one reader being slow does not stop another.
Key Concepts
1
A Kafka cluster is a set of brokers (servers). Data is organised into topics, and every topic is split into partitions — the unit of parallelism and ordering. Each partition is an ordered, immutable sequence of records stored on disk. Historically a ZooKeeper ensemble managed cluster metadata and controller election; modern Kafka (KRaft mode) folds that role into the brokers themselves, removing the ZooKeeper dependency.
2
The design favours high throughput: sequential disk writes, zero-copy transfers to the network, batching, and compression let a single cluster handle millions of messages per second.
3
What the interviewer is probing.
What the interviewer is probing.
4
1. "What is a partition and why does it exist?" *Probing:* the unit of parallelism. *Stalls:* "A
piece of a topic." *Moves up:* it is the unit of ordering, storage and parallelism — consumers scale
to the partition count, which is why that number is a capacity decision.
1. "What is a partition and why does it exist?"
5
2. "What replaced ZooKeeper and why does it matter?" *Probing:* currency. *Stalls:* "Nothing
changed." *Moves up:* KRaft mode puts metadata in Kafka itself, removing a second system to operate
and raising the practical partition ceiling.
2. "What replaced ZooKeeper and why does it matter?"
6
3. "Why is Kafka fast despite writing to disk?" *Probing:* the mechanism. *Stalls:* "It is in
memory." *Moves up:* append-only sequential writes and the page cache, with zero-copy transfer to
consumers — sequential disk is not the bottleneck people assume.
3. "Why is Kafka fast despite writing to disk?"
7
4. "What does the broker not do that a queue broker does?" *Probing:* the distinction. *Stalls:*
"Nothing." *Moves up:* it does not track per-message acknowledgement or remove messages on read —
consumers track their own offset and data is retained by policy.
4. "What does the broker not do that a queue broker does?"