Skip to content
Kafka Is FunProgress

CriticalCluster Metadata

Controller Election Loop

Symptoms

  • Controller epoch rapidly incrementing
  • Frequent LeaderAndIsrRequest storms in broker logs
  • Topic creation/deletion operations timing out
  • General cluster instability despite all brokers being online

Root cause

The controller keeps failing and being re-elected. Can be caused by: controller broker GC pauses causing ZooKeeper session timeout, metadata corruption, or network issues specific to the controller broker. Each election increments the epoch and triggers a metadata reconciliation that can overwhelm brokers.

Detection

kafka.controller:ActiveControllerCount fluctuates between 0 and 1. Controller epoch in ZooKeeper/KRaft metadata incrementing every few seconds. High rate of LeaderAndIsrRequest in broker logs.

Immediate actions

  1. Identify the current controller: kafka-metadata-quorum.sh or ZK: get /controller
  2. Check controller broker for GC pauses, CPU spikes, disk I/O issues
  3. Review controller.log for exception stack traces
  4. Consider restarting the problematic controller broker

Permanent fix

Fix controller broker resource issues. In ZooKeeper mode: increase ZooKeeper session timeout (zookeeper.session.timeout.ms). In KRaft: use dedicated controller nodes (separate from brokers) to prevent IO from interfering with Raft heartbeats.

Prevention

Dedicated KRaft controller nodes. Small JVM heap on controllers (4-6GB) to minimize GC pauses. Monitor controller epoch rate of change. Use ZGC or Shenandoah for controller JVM.

Reproduce it in the simulator

Run Network Partition scenario — observe B2 (Controller) losing controller status and B3 winning election. Controller epoch increments (visible in the epoch badge).

The interactive simulator is being migrated to /simulate.

The interview angle

Rapidly incrementing controller epoch is a red flag. In KRaft, use dedicated controller quorum nodes. In ZooKeeper mode, tune zookeeper.session.timeout.ms above your 99th percentile GC pause.