Skip to content
Kafka Is FunProgress

HighReplication Degradation

ISR Shrink / Under-Replicated Partitions

Symptoms

  • UnderReplicatedPartitions metric > 0
  • Partition ISR lists shorter than replication factor
  • Replica fetch lag growing for specific brokers
  • Increased recovery time after subsequent failures

Root cause

One or more follower replicas can no longer keep up with the leader within replica.lag.time.max.ms. Common causes: broker GC pauses, slow disk I/O, network congestion between brokers, high CPU from other workloads. The broker is removed from ISR, reducing fault tolerance.

Detection

kafka.server:UnderReplicatedPartitions > 0. kafka.server:ReplicaFetcherManager:MaxLag per topic-partition. Follower LEO diverging from leader LEO.

Immediate actions

  1. Identify which broker is lagging: kafka-log-dirs.sh --describe --bootstrap-server
  2. Check lagging broker for: GC pauses (gc.log), disk I/O saturation (iostat), CPU spikes
  3. Check network bandwidth between lagging broker and leader: iperf3
  4. Monitor if ISR is auto-recovering (URP trending down) or stuck

Permanent fix

Address the root cause (GC tuning, disk upgrade, network fix). The replica will automatically rejoin ISR once its lag drops below replica.lag.time.max.ms.

Prevention

Tune JVM GC (G1GC with small pause target), use NVMe SSDs for broker storage, ensure dedicated network for inter-broker replication, avoid running other I/O-intensive workloads on broker nodes.

Reproduce it in the simulator

In Simulator Cluster view: drag any broker's lag slider above 5s. Watch it get removed from ISR. Drag back to 0 — watch it rejoin.

The interactive simulator is being migrated to /simulate.

The interview angle

ISR shrink without a crash is subtle and dangerous. If ISR=2 and another broker crashes, you're at ISR=1 < minISR=2 and writes start failing. Monitor URP continuously.