Storage Internals
How Kafka stores data on disk — segments, indexes, compaction, and retention.
- 1The Append-Only LogEach Kafka partition is stored as a sequence of segment files on disk. Kafka only appends — it never modifies or deletes…
- 2Segment Files & Index FilesEach segment consists of three files: .log (the actual messages), .index (sparse offset-to-byte-position index), and .ti…
- 3Message RetentionKafka supports two retention strategies: time-based (delete messages older than retention.ms) and size-based (delete old…
- 4Log CompactionLog compaction is an alternative to deletion. Instead of time/size-based deletion, Kafka keeps only the LATEST message p…
- 5Batching & CompressionProducers batch messages before sending to reduce network overhead and improve throughput. linger.ms controls how long t…
Knowledge check
Q1.What does log compaction guarantee?
Q2.Why are Kafka's sequential writes so fast?
Q3.What determines when a segment file can be deleted?