Apache Kafka
Log Retention & Compaction
Decide how long data lives — by time/size, or by keeping only the latest value per key.
Kafka keeps messages independently of whether they have been consumed; retention policy decides when they age out. Time/size retention deletes whole log segments once they exceed retention.ms or retention.bytes — ideal for event streams where old events stop mattering.
Time retention is a security camera that overwrites footage older than 30 days. Compaction is an address book: you only keep each person's current phone number, discarding old numbers, but you keep every person forever.
Key Concepts
1
Log compaction is different: instead of deleting by age, it guarantees that the latest value for each key is always retained, while older values for the same key are eventually garbage-collected. This turns a topic into a durable, replayable changelog of "current state per key" — the basis of Kafka's compacted topics, Kafka Streams state stores, and __consumer_offsets.
2
A topic can combine both: compaction plus a time floor. A delete is represented by a tombstone (a record with a null value), which compaction eventually removes after a configurable delay.
3
What the interviewer is probing.
What the interviewer is probing.
4
1. "What is the difference between retention and compaction?" *Probing:* the distinction.
*Stalls:* "Both delete." *Moves up:* retention drops records older than a time or size; compaction
keeps the latest value per key indefinitely.
1. "What is the difference between retention and compaction?"
5
2. "When would you use a compacted topic?" *Probing:* the fit. *Stalls:* "To save space." *Moves
up:* when the topic is a changelog of current state — replaying it rebuilds the latest value for
every key.
2. "When would you use a compacted topic?"
6
3. "How is a delete represented in a compacted topic?" *Probing:* the detail. *Stalls:* "It is
removed." *Moves up:* a tombstone — a null value for the key, retained long enough for consumers to
see it before being cleaned.
3. "How is a delete represented in a compacted topic?"
7
4. "What is the risk of a long retention?" *Probing:* the practical angle. *Stalls:* "Disk."
*Moves up:* storage cost and a very long replay time for a new consumer, which turns a routine
redeploy into an incident.
4. "What is the risk of a long retention?"