Storage Internals
How Kafka stores data on disk — segments, indexes, compaction, and retention.
0 of 5 lessons complete
- 1The Append-Only LogEach partition is stored as a sequence of segment files on disk. Writes only ever append to the end of the active segmen…
- 2Segment Files & Index FilesA partition's directory holds segment files named by the base offset of the first record they contain, for example 00000…
- 3Message RetentionKafka deletes data one log segment at a time, using time and size policies. Time-based retention (retention.ms) is judge…
- 4Log CompactionLog compaction is an alternative or a complement to deletion: cleanup.policy accepts compact, delete, or both. It guaran…
- 5Batching & CompressionProducers batch records before sending, which spreads the overhead of a network round trip over many records. linger.ms …
Knowledge check
Q1.What does log compaction guarantee?
Q2.Why are Kafka's appends to disk fast?
Q3.With time-based retention, when can a segment file be deleted?