HighConsumer Group
Rebalance Storm
Symptoms
- Group members keep leaving and rejoining (kafka-consumer-groups.sh --describe --members shows the membership changing)
- Lag grows while the group keeps rebalancing
- Commits fail with CommitFailedException because the group has already moved the partition elsewhere
Detection
kafka-consumer-groups.sh --describe --members --group <group> shows the members and how many partitions each has. Repeated CommitFailedException and members that keep disappearing are the signs.
Challenge mode: diagnose the root cause from the symptoms before reading the answer.
Root cause
A rebalance runs when a member joins, leaves or fails. A member is also removed if it sends no heartbeat within session.timeout.ms (45000 ms by default, classic protocol) or does not call poll() within max.poll.interval.ms (300000 ms by default); it then leaves the group so another consumer can take over. A storm is those events repeating: processing that takes longer than the poll interval, consumers that keep restarting, or pauses long enough to miss heartbeats.
Immediate actions
- Compare max.poll.interval.ms with how long a batch from poll() really takes to process
- Lower max.poll.records so each poll() batch finishes sooner
- Check for pauses long enough to miss heartbeats (session.timeout.ms)
- If consumers run in containers, check how often they restart and why
Permanent fix
Static membership: give each consumer a unique, stable group.instance.id and a larger session.timeout.ms, so a consumer that restarts is not treated as a new member (not supported with group.protocol=consumer). The CooperativeStickyAssignor revokes only the partitions that have to move instead of everything.
Prevention
Keep processing time per poll inside max.poll.interval.ms, use a smaller max.poll.records if needed, and consider group.instance.id and the CooperativeStickyAssignor.
Reproduce it in the simulator
In the simulator's Consumer tab, add and remove consumers quickly. Each change triggers a rebalance: watch the Rebalancing badge appear and the assignment change. Switch the group protocol to Consumer (KIP-848) to compare.
Open the simulator to try it.
The interview angle
Know what triggers a rebalance and which settings stop a healthy consumer from being removed (max.poll.interval.ms, session.timeout.ms), and know that static membership and cooperative rebalancing reduce the disruption.
Sources
- Consumer configs (4.2 documentation) (opens in a new tab)
- KafkaConsumer Javadoc (4.2) (opens in a new tab)
- KIP-345: Static membership protocol (opens in a new tab)
- KIP-429: Kafka Consumer Incremental Rebalance Protocol (opens in a new tab)
- Basic Kafka operations (4.2 documentation) (opens in a new tab)
Checked against Kafka 4.2 documentation on 11 October 2026. Draft: not yet reviewed by a Kafka specialist, and releases move on, so confirm against the current documentation before you act on it.
Seeing this in your logs?
- CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another memberYour processing between two calls to poll() was longer than max.poll.interval.ms (300000 ms, five minutes, by default). The exception message itself suggests either increasing max.poll.interval.ms or reducing the batch size with max.poll.records (500 by default).
- Consumer group keeps rebalancingFind out why members keep leaving: slow polling (max.poll.interval.ms), missed heartbeats, or restarts and scaling. Then make each rebalance cheaper with static membership, a cooperative assignor, or the new consumer protocol.