Log Compaction

Overview

Log compaction is an alternative or a complement to deletion: cleanup.policy accepts compact, delete, or both. It guarantees that Kafka retains at least the last known value for each message key. Older values for a key are removed in the background, and the head of the log, which has not been compacted yet, keeps all messages. A record with a key and a null payload is a tombstone: it is treated as a delete and is cleaned out of the log after a period of time. The log therefore holds a snapshot of the latest value for every key, so consumers can rebuild state by reading from the start, but intermediate history is not kept. Kafka Connect in distributed mode stores connector configs, offsets and statuses in compacted topics, and Kafka Streams enables compaction on the changelog topics behind its state stores.

Analogy

Key configuration

cleanup.policy=compact — enable compaction (broker: log.cleanup.policy)
min.cleanable.dirty.ratio=0.5 — default; the dirty ratio at which a log becomes eligible for cleaning (broker: log.cleaner.min.cleanable.ratio)
delete.retention.ms=86400000 — default, 1 day: how long tombstones are kept (broker: log.cleaner.delete.retention.ms)

Production tip

Sources

• Design (4.2 documentation) (opens in a new tab)
• Topic configs (4.2 documentation) (opens in a new tab)
• Kafka Connect user guide (4.2 documentation) (opens in a new tab)
• Streams architecture (4.2 documentation) (opens in a new tab)

Checked against Kafka 4.2 documentation on Oct 2026. Draft: not yet reviewed by a Kafka specialist. Releases move on, so confirm against the current documentation before you act on a number.