When a failed broker restarts, it truncates its local log to the last known HWM (removing potentially divergent messages), then starts replicating from the leader. It is added back to the ISR only after it has fully caught up (its LEO matches the leader's LEO within the replica.lag.time.max.ms threshold). This process is automatic — no manual intervention required.
Lesson 5 of 5
Replica Recovery Process
Try it in the simulator
Guided scenarios for this module — the simulator opens with the exercise already running.
- Break a broker, watch the failoverStop a partition leader yourself, watch the election, prove producing still works, then heal the cluster.Intermediate · ~6 min
- Shrink the ISR with follower lagDrag a follower's replication lag past the threshold, watch it ejected from the ISR, then let it recover.Intermediate · ~5 min
- Trigger NOT_ENOUGH_REPLICASKill two brokers to violate min.insync.replicas and watch Kafka refuse the write rather than risk data loss.Advanced · ~6 min