Apache Kafka
Log Retention & Compaction
Decide how long data lives — by time/size, or by keeping only the latest value per key.
Kafka keeps messages independently of whether they have been consumed; retention policy decides when they age out. Time/size retention deletes whole log segments once they exceed retention.ms or retention.bytes — ideal for event streams where old events stop mattering.
Time retention is a security camera that overwrites footage older than 30 days. Compaction is an address book: you only keep each person's current phone number, discarding old numbers, but you keep every person forever.
Key Concepts
1
Log compaction is different: instead of deleting by age, it guarantees that the latest value for each key is always retained, while older values for the same key are eventually garbage-collected. This turns a topic into a durable, replayable changelog of "current state per key" — the basis of Kafka's compacted topics, Kafka Streams state stores, and __consumer_offsets.
2
A topic can combine both: compaction plus a time floor. A delete is represented by a tombstone (a record with a null value), which compaction eventually removes after a configurable delay.