Apache Kafka
Architecture Overview
Understand Kafka as a distributed, partitioned, replicated commit log — not a traditional queue.
Kafka is best understood as a distributed append-only log rather than a message queue. Producers append records to the end of a log; consumers read forward at their own pace by tracking an offset. Because messages are not deleted when read, many independent consumers can replay the same data, and a slow consumer never blocks a fast one.
A newspaper archive: every edition is filed in order and never thrown away. Any number of readers can start from any past date and read forward at their own speed — one reader being slow does not stop another.
Key Concepts
1
A Kafka cluster is a set of brokers (servers). Data is organised into topics, and every topic is split into partitions — the unit of parallelism and ordering. Each partition is an ordered, immutable sequence of records stored on disk. Historically a ZooKeeper ensemble managed cluster metadata and controller election; modern Kafka (KRaft mode) folds that role into the brokers themselves, removing the ZooKeeper dependency.
2
The design favours high throughput: sequential disk writes, zero-copy transfers to the network, batching, and compression let a single cluster handle millions of messages per second.