Apache Kafka
Consumer Groups & Rebalancing
Scale consumption horizontally while each message is processed once per group.
Consumers join a consumer group identified by a group.id. Kafka assigns each partition of a subscribed topic to exactly one consumer in the group, so the group as a whole processes every partition in parallel without double-processing. Different groups are independent and each receive the full stream.
A team of cashiers (one group) splitting the checkout lanes (partitions) of one store: each lane is staffed by exactly one cashier. A second, separate team auditing the same lanes is a different group that sees every transaction independently.
Key Concepts
1
The practical ceiling on parallelism is the partition count: if a topic has 6 partitions, at most 6 consumers in a group do useful work; extra consumers sit idle. When members join, leave, or crash, the group coordinator triggers a rebalance to redistribute partitions. Modern Kafka uses cooperative/incremental rebalancing to avoid the old "stop-the-world" pause.
2
Consumers commit offsets (auto or manual) to record progress, so a restarted consumer resumes where the group left off.
3
What the interviewer is probing.
What the interviewer is probing.
4
1. "How does work divide across a consumer group?" *Probing:* the model. *Stalls:* "Evenly."
*Moves up:* each partition goes to exactly one consumer in the group, so more consumers than
partitions leaves some idle.
1. "How does work divide across a consumer group?"
5
2. "What triggers a rebalance and why is it costly?" *Probing:* the operational pain. *Stalls:*
"A restart." *Moves up:* a member joining, leaving or timing out — and during it consumption stops,
so frequent rebalances look like an outage.
2. "What triggers a rebalance and why is it costly?"
6
3. "How do you reduce rebalance impact?" *Probing:* the tuning. *Stalls:* "Fewer consumers."
*Moves up:* cooperative sticky assignment, static group membership, and a session timeout tuned
against realistic processing time.
3. "How do you reduce rebalance impact?"
7
4. "Why does a slow consumer get kicked out?" *Probing:* the subtle cause. *Stalls:* "Network."
*Moves up:* max.poll.interval.ms elapsed between polls, so long processing inside the loop looks
like a dead member.
4. "Why does a slow consumer get kicked out?"