CriticalNetwork Failure
Network Partition
Symptoms
- Brokers drop out of the cluster while their processes are still running: the controller stops hearing their heartbeats and the Fenced Broker Count metric rises
- UnderReplicatedPartitions rises as the cut-off broker is removed from ISRs
- Clients that were talking to the cut-off broker see errors or timeouts
Detection
The controller's Fenced Broker Count metric and UnderReplicatedPartitions. kafka-metadata-quorum.sh describe --status shows the quorum leader and its epoch, and describe --replication shows the state of each controller.
Challenge mode: diagnose the root cause from the symptoms before reading the answer.
Root cause
A network partition cuts some brokers off from the others, or from the controller, while their processes keep running. Brokers keep a session with the controller by sending heartbeats; a broker that sends none before broker.session.timeout.ms (9000 ms by default) is considered offline. The controller fences it, removes it from the ISRs, and elects new leaders from the remaining ISR members. Kafka's documentation says the cluster stays available through node failures after a short fail-over, but may not stay available through network partitions.
Immediate actions
- Find out which brokers cannot reach each other, or cannot reach the controller
- Check which partitions lost in-sync replicas (UnderReplicatedPartitions, kafka-topics.sh --describe)
- With acks=all, a write is refused once a partition's ISR is smaller than min.insync.replicas: the producer gets NotEnoughReplicas, which it retries
- Look at the controller quorum: a change of leader or a rising epoch in describe --status means the controllers lost contact with their leader
Permanent fix
Restore the network. The broker's heartbeats resume, the controller unfences it once its metadata is reasonably current, and its replicas must fully re-sync with their leaders before they rejoin the ISRs.
Prevention
Spread replicas across racks or availability zones with broker.rack. Use acks=all with min.insync.replicas so a partition refuses writes rather than accept them on too few replicas. Run 3 or 5 controllers.
Reproduce it in the simulator
In the simulator, run the Network Partition demo to watch the controller (Broker 2) get cut off, its session expire (about 2 s here, 9 s by default in Kafka), another broker take over as controller with a higher epoch, and Broker 2 come back as a follower. Press Isolate on a broker (Cluster tab, or Cluster controls on the Overview tab) to see its partitions get new leaders and the broker leave the ISRs; Isolate does not move the controller. Reconnect it and watch it catch up.
Open the simulator to try it.
The interview angle
Say what Kafka documents: it stays available through node failures after a short fail-over but may not through network partitions, and with acks=all and min.insync.replicas a partition refuses writes rather than accept them on too few replicas.
Sources
- Design (4.2 documentation) (opens in a new tab)
- Broker configs (4.2 documentation) (opens in a new tab)
- KRaft operations (4.2 documentation) (opens in a new tab)
- Monitoring (4.2 documentation) (opens in a new tab)
- KIP-631: The Quorum-based Kafka Controller (opens in a new tab)
- KIP-595: A Raft Protocol for the Metadata Quorum (opens in a new tab)
- Basic Kafka operations (4.2 documentation) (opens in a new tab)
Checked against Kafka 4.2 documentation on 11 October 2026. Draft: not yet reviewed by a Kafka specialist, and releases move on, so confirm against the current documentation before you act on it.