Skip to content
Kafka Is FunProgress

HighBroker Failure

Leader Broker Crash

Symptoms

  • LEADER_NOT_AVAILABLE errors in producer logs
  • NOT_LEADER_OR_FOLLOWER exceptions
  • Consumer lag spike on affected partitions
  • UnderReplicatedPartitions metric jumps

Root cause

The partition's leader broker process terminated (OOM, hardware failure, OS crash, kernel panic, or manual kill). All produce/consume requests for that partition fail until a new leader is elected from the ISR.

Detection

JMX: kafka.server:UnderReplicatedPartitions > 0. Controller logs show LeaderAndIsrRequest for affected partitions. Producer error counters spike.

Immediate actions

  1. Verify broker is actually down (not just paused): check broker logs, process list
  2. Check if ISR is healthy — new leader should be elected automatically within seconds
  3. Monitor UnderReplicatedPartitions — should drop to 0 after election
  4. Check producer retry behavior — with enable.idempotence=true, retries are safe

Permanent fix

Start the failed broker. It will automatically truncate to last HWM, replicate missed messages, and rejoin ISR. No manual intervention needed for healthy clusters.

Prevention

RF=3, minISR=2, enable.idempotence=true on producers. Set up alerts on URP > 0 and broker liveness. Use systemd watchdog or K8s pod restart policies.

Reproduce it in the simulator

In the Simulator: click 'Stop' on any broker. Observe leader election in event log. Try sending a message — see it fail then recover. Click 'Start' to bring it back.

The interactive simulator is being migrated to /simulate.

The interview angle

Key insight: Kafka auto-recovers from leader failures without manual intervention IF ISR is healthy. The danger is when ISR is already degraded before the crash.