CriticalCluster Metadata
Controller Election Loop
Symptoms
- The quorum leader epoch in kafka-metadata-quorum.sh describe --status keeps rising
- The controller's ActiveControllerCount keeps changing as the active controller changes
- The Fenced Broker Count metric rises as brokers lose their session with the controller
Detection
kafka-metadata-quorum.sh describe --status shows the LeaderId and LeaderEpoch. The monitoring page lists the active-controller metric kafka.controller:type=KafkaController,name=ActiveControllerCount (0 or 1) and the raft metrics Current Leader and Current Epoch.
Challenge mode: diagnose the root cause from the symptoms before reading the answer.
Root cause
The metadata quorum keeps electing new leaders. A controller that gets no fetch response from the quorum leader within controller.quorum.fetch.timeout.ms (2000 ms by default) starts an election by bumping the leader epoch and voting for itself, and a leader that cannot reach a majority of the quorum resigns. A controller that is slow or overloaded, or on an unstable network, can therefore trigger a run of elections.
Immediate actions
- Find the current quorum leader: kafka-metadata-quorum.sh describe --status
- Check the controller nodes' logs, CPU, disk and network, and the network between them
- If controllers share a process with brokers (combined mode), check whether broker load is starving them
Permanent fix
Fix what makes the controller unreachable or slow. Run the controllers on their own nodes: the documentation says combined mode is not recommended in critical deployments, and in it you cannot roll or scale controllers separately from brokers.
Prevention
Set process.roles to either broker or controller, not both. Use 3 or 5 controllers (a quorum of 2N+1 withstands N failures). Watch how often the epoch changes.
Reproduce it in the simulator
Run the Network Partition demo: Broker 2 (the controller) is cut off, another broker becomes controller, and the epoch badge goes up.
Open the simulator to try it.
The interview angle
A rapidly rising epoch is a red flag. In KRaft each leader election increments the quorum epoch. Dedicated controller nodes keep brokers' load away from the metadata quorum.
Sources
- KRaft operations (4.2 documentation) (opens in a new tab)
- Monitoring (4.2 documentation) (opens in a new tab)
- Broker configs (4.2 documentation) (opens in a new tab)
- QuorumConfig.java (4.2 source) (opens in a new tab)
- KIP-595: A Raft Protocol for the Metadata Quorum (opens in a new tab)
- KIP-631: The Quorum-based Kafka Controller (opens in a new tab)
Checked against Kafka 4.2 documentation on 11 October 2026. Draft: not yet reviewed by a Kafka specialist, and releases move on, so confirm against the current documentation before you act on it.