HighConsumer Failure
Consumer Lag Explosion
Symptoms
- Consumer lag keeps growing instead of shrinking
- The group still has all its members, but lag does not fall
- Records are processed more slowly than they are produced
Detection
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group <group> prints CURRENT-OFFSET, LOG-END-OFFSET and LAG (the log end offset minus the current offset) for each partition. Run it a few times: is the lag stable, growing or shrinking?
Challenge mode: diagnose the root cause from the symptoms before reading the answer.
Root cause
The group is reading more slowly than records are being written. Typical causes: slow processing between polls, too few consumers for the partitions, repeated errors on the same records, or one partition that is much busier than the others. Each partition goes to exactly one consumer in a group, so the partition count limits how many consumers can share the work.
Immediate actions
- Check the consumers' logs for errors or slow processing
- Check what the consumer does between polls: if it takes longer than max.poll.interval.ms (300000 ms by default) the consumer is removed from the group
- Add consumers, but only up to the partition count: extra consumers get no partitions
- If one record keeps failing, set it aside so the partition can move on (in Spring Kafka, a dead-letter topic)
Permanent fix
Make the processing faster, add consumers up to the partition count, or add partitions if you need more consumers than you have. Handle records that cannot be processed separately instead of retrying them forever.
Prevention
Alert on how lag trends over time, with a threshold chosen for your own service level (Apache gives none). Keep the time between polls well inside max.poll.interval.ms, adjusting max.poll.records (500 by default) if a batch takes too long.
Reproduce it in the simulator
In the simulator's Offsets and Consumer tabs, send several messages and watch each group's lag and offsets.
Open the simulator to try it.
The interview angle
Whether lag is a temporary spike or a sustained problem shows in its trend: is it growing, stable or shrinking? And what limits adding consumers: the partition count.
Sources
- Basic Kafka operations (4.2 documentation) (opens in a new tab)
- Consumer configs (4.2 documentation) (opens in a new tab)
- KafkaConsumer Javadoc (4.2) (opens in a new tab)
- ConsumerGroupCommand.java (4.2 source) (opens in a new tab)
- Design (4.2 documentation) (opens in a new tab)
- Handling exceptions in @KafkaListener (Spring for Apache Kafka) (opens in a new tab)
Checked against Kafka 4.2 documentation on 11 October 2026. Draft: not yet reviewed by a Kafka specialist, and releases move on, so confirm against the current documentation before you act on it.