Skip to content
Kafka Is FunProgress

CriticalNetwork Failure

Network Partition (Split-Brain)

Symptoms

  • Consumer lag growing on isolated partition's topics
  • Two brokers both reporting as leader for same partition
  • Intermittent 'outdated epoch' errors in logs
  • MetaData cache invalidation loops in producers

Root cause

A network partition severs communication between some brokers while all processes remain alive. The isolated broker believes it's still the leader; the rest of the cluster elects a new leader. This creates a split-brain window where two brokers may briefly accept writes for the same partition.

Detection

Monitor inter-broker replication lag. Watch for controller epoch changes. Alert on brokers with no heartbeat received despite being 'alive' according to process monitoring.

Immediate actions

  1. Identify the partition boundary — which brokers can't reach each other
  2. Do NOT manually interfere with leader elections during partition
  3. Producers with acks=all are protected — they require ISR quorum
  4. Check if any consumers are reading from the isolated (stale) broker

Permanent fix

Restore network connectivity. The isolated broker will detect the new controller epoch, step down as leader (fenced by higher epoch), truncate divergent data, and rejoin ISR.

Prevention

Deploy brokers across availability zones with redundant network paths. Use acks=all + minISR=2 so writes require acknowledgement from multiple AZs. Enable rack awareness (broker.rack).

Reproduce it in the simulator

In the Simulator: click 'Isolate' on Broker 2 (Controller). Watch ZK session timeout, new Controller election, and B2 isolation badge. Reconnect after a few seconds.

The interactive simulator is being migrated to /simulate.

The interview angle

Network partition is the hardest Kafka failure mode. Key protection: acks=all ensures writes need ISR quorum across the network partition boundary, preventing divergent writes.