HighBroker Failure
Leader Broker Crash
Symptoms
- Produce and fetch requests for the partitions the broker led fail with retriable errors such as NOT_LEADER_OR_FOLLOWER or LEADER_NOT_AVAILABLE until a new leader is known
- UnderReplicatedPartitions rises above 0 and stays there while the broker is down
- Clients refresh their metadata before retrying
Detection
kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions is above 0 (its normal value is 0). kafka-topics.sh --describe --under-replicated-partitions lists the affected partitions.
Challenge mode: diagnose the root cause from the symptoms before reading the answer.
Root cause
The broker that led one or more partitions stopped: a process crash, a hardware failure or a kill. Until the controller elects a new leader from each partition's in-sync replicas, requests for those partitions fail.
Immediate actions
- Confirm the broker process is really down and not just slow: check its logs and the process list
- Check that each affected partition has a new leader: kafka-topics.sh --describe shows the leader and the ISR
- Expect UnderReplicatedPartitions to stay above 0 until the broker is back and its replicas have caught up; electing a new leader does not clear it
- Producers retry on their own: retries default to 2147483647, bounded by delivery.timeout.ms (120000 ms by default), and with enable.idempotence=true (the default) a retry within the same producer session does not write a duplicate
Permanent fix
Start the failed broker. As a follower it removes any entries that differ from the leader's log, fetches what it missed, and rejoins the ISR once it has fully caught up. No manual step is needed for the partitions to recover.
Prevention
Use a replication factor above 1 and set min.insync.replicas (the documentation's example is a replication factor of 3 with min.insync.replicas=2 and acks=all). Alert on UnderReplicatedPartitions that stays above 0.
Reproduce it in the simulator
In the simulator, stop a broker (Cluster tab, or Cluster controls on the Overview tab) and read the event log: a new leader is chosen from the ISR. Send a message to a partition that broker led and watch the first attempt fail and the retry succeed. Start the broker again to see it catch up and rejoin the ISR.
Open the simulator to try it.
The interview angle
A new leader can only come from the ISR while unclean leader election is off (the default). If the ISR held only the leader, no in-sync replica is left, and the partition stays unavailable until one returns.
Sources
- Design (4.2 documentation) (opens in a new tab)
- Protocol and error codes (4.2 documentation) (opens in a new tab)
- Producer configs (4.2 documentation) (opens in a new tab)
- Monitoring (4.2 documentation) (opens in a new tab)
- Broker configs (4.2 documentation) (opens in a new tab)
- AbstractFetcherThread.scala (4.2 source) (opens in a new tab)
Checked against Kafka 4.2 documentation on 11 October 2026. Draft: not yet reviewed by a Kafka specialist, and releases move on, so confirm against the current documentation before you act on it.
Seeing this in your logs?
- TimeoutException: Expiring 5 record(s) for orders-0:120000 ms has passed since batch creationCheck that the leader of the partition named in the message is up and reachable from the producer, and look at how fast the broker is accepting data. Also make sure your send callback or future handles the failure, because otherwise nothing else tells you these records were not delivered.
- Under-replicated partitions greater than zeroLook at which broker the missing replicas live on. If it is down, bring it back. If it is up, a follower is not keeping up, and the fix is whatever slows it. Also check whether one of its log directories has failed.