AdvancedInternals Deep Dive
How does Kafka handle log truncation during replica recovery?
Reference answer
When a broker restarts after failure, it may have uncommitted messages that the leader never counted in HWM (written to follower LEO but not all ISR). The broker truncates its log to the last HWM it received before failure. This is safe because the leader's HWM represents the highest offset replicated to all ISR. Any messages above the follower's last known HWM might conflict with the leader's authoritative state. After truncation, the follower starts fetching from the leader to catch up. This ensures all ISR replicas have identical committed logs.
Expected key concepts: log truncation, uncommitted, HWM, last known HWM, follower LEO, leader, ISR, identical committed logs, restart, authoritative