Apache Kafka
Consumer Groups & Rebalancing
Scale consumption horizontally while each message is processed once per group.
Consumers join a consumer group identified by a group.id. Kafka assigns each partition of a subscribed topic to exactly one consumer in the group, so the group as a whole processes every partition in parallel without double-processing. Different groups are independent and each receive the full stream.
A team of cashiers (one group) splitting the checkout lanes (partitions) of one store: each lane is staffed by exactly one cashier. A second, separate team auditing the same lanes is a different group that sees every transaction independently.
Key Concepts
1
The practical ceiling on parallelism is the partition count: if a topic has 6 partitions, at most 6 consumers in a group do useful work; extra consumers sit idle. When members join, leave, or crash, the group coordinator triggers a rebalance to redistribute partitions. Modern Kafka uses cooperative/incremental rebalancing to avoid the old "stop-the-world" pause.
2
Consumers commit offsets (auto or manual) to record progress, so a restarted consumer resumes where the group left off.