The Append-Only Log

Overview

Each partition is stored as a sequence of segment files on disk. Writes only ever append to the end of the active segment. Under the delete policy, data is removed a whole segment at a time, and under compaction the cleaner recopies segments in the background. Appending sequentially lets Kafka rely on the operating system's read-ahead and write-behind, and Kafka relies on the page cache instead of keeping its own cache. The design page cites linear writes of about 600MB/sec against about 100kB/sec for random writes on one array of spinning disks.

Analogy

Key configuration

log.dirs=/var/kafka/logs — data directories, comma-separated (log.dir is the supplemental single-directory setting, default /tmp/kafka-logs)
log.segment.bytes=1073741824 — 1 GiB per segment (default)
log.roll.ms — time-based segment roll (null by default, then log.roll.hours = 168 applies)

Production tip

Sources

• Log implementation (4.2 documentation) (opens in a new tab)
• Design (4.2 documentation) (opens in a new tab)
• Broker configs (4.2 documentation) (opens in a new tab)
• Hardware and OS (4.2 documentation) (opens in a new tab)

Checked against Kafka 4.2 documentation on Oct 2026. Draft: not yet reviewed by a Kafka specialist. Releases move on, so confirm against the current documentation before you act on a number.