What is the High Watermark (HWM)?

The high watermark marks how far a partition's log is committed. For a normal (read_uncommitted) consumer the end offset is the high watermark, defined as the offset of the last successfully replicated message plus one, so it is the offset just past the last replicated record. A write is committed only when all in-sync replicas have received it, and only committed messages are given out to consumers, so consumers can read records below the high watermark. This protects consumers from reading data that could be lost if the leader fails. The leader advances the high watermark as followers fetch and passes it to followers in fetch responses. The log end offset (LEO) is the offset of the next message that will be appended to a replica's log, so on the leader the records between the HWM and the LEO are written but not yet replicated to all in-sync replicas. A read_committed consumer is bounded by the last stable offset (LSO), which also depends on open transactions.

Expected key concepts: high watermark, committed, in-sync replicas, last successfully replicated message plus one, consumers, log end offset, LEO, leader fails, fetch responses, last stable offset