What is log compaction and when would you use it?

Log compaction ensures Kafka always retains at least the last known value for each message key within a partition's log. Older records with the same key can be removed; compaction never reorders messages, it only removes some. A record with a key and a null value is a tombstone, a delete marker for that key; tombstones are themselves cleaned out after a period, controlled by delete.retention.ms (default 1 day). Unlike time or size retention, which deletes whole segments, compaction keeps the latest record per key, and the active segment is not compacted. It is enabled with cleanup.policy=compact on a topic (log.cleanup.policy at the broker), and cleanup.policy=compact,delete combines compaction with deletion. The Apache docs name these uses: subscribing to a database's change feed (CDC), event sourcing, and journaling for high availability. It suits any topic where the latest value per entity is the state that matters.

Expected key concepts: latest value, key, tombstone, delete.retention.ms, CDC, event sourcing, cleanup.policy, compact, delete, active segment