Skip to content
Kafka Is Fun

HighConsumer Group

Rebalance Storm

Symptoms

  • Constant 'Rebalance triggered' in consumer logs
  • Consumer lag growing despite consumers being 'active'
  • max.poll.interval.ms exceeded warnings
  • Processing throughput near zero during rebalances

Detection

Consumer logs show rapid consecutive JoinGroupRequest/SyncGroupRequest. Lag grows despite consumers being assigned. Member count in group fluctuates rapidly.

Challenge mode: diagnose the root cause from the symptoms before reading the answer.

Root cause

Consumers are repeatedly triggering rebalances faster than they can complete. Common causes: max.poll.interval.ms set too low (consumer spends too long processing and gets kicked out, triggering rebalance), rapid pod restarts in Kubernetes, or session.timeout.ms too short relative to GC pauses.

Immediate actions

  1. Check max.poll.interval.ms vs actual processing time per batch
  2. Reduce max.poll.records so each poll() completes faster
  3. Check for GC pauses causing heartbeat interruptions
  4. In K8s: check pod restart frequency and reasons

Permanent fix

Switch to group.instance.id (static membership) — pods that restart within session.timeout.ms don't trigger rebalance. Use CooperativeStickyAssignor to minimize disruption.

Prevention

group.instance.id in K8s, CooperativeStickyAssignor, max.poll.interval.ms > worst-case processing time, reduce heap to reduce GC pauses.

Reproduce it in the simulator

In Consumer view: rapidly add and remove consumers. Each change triggers rebalance. Watch the 'Rebalancing...' banner appear and disappear.

The interactive simulator is being migrated to /simulate.

The interview angle

Rebalance storms are a K8s Kafka anti-pattern. The fix is always: group.instance.id + CooperativeStickyAssignor + appropriate session/poll timeouts.