Skip to content
Kafka Is FunProgress

HighConsumer Failure

Consumer Lag Explosion

Symptoms

  • Consumer lag metric growing continuously
  • Consumer group health shows all members but lag not decreasing
  • Processing throughput much lower than production rate
  • Potential downstream SLA breaches

Root cause

Consumer group is processing slower than messages are being produced. Could be: processing logic too slow (CPU, DB call, API call), too few consumers, an exception causing repeated processing attempts, or a single slow partition causing head-of-line blocking.

Detection

kafka-consumer-groups.sh --describe shows LAG per partition. Check if lag is stable (consuming = producing) or growing (consuming < producing). Check consumer logs for exceptions or slow processing warnings.

Immediate actions

  1. Check consumer logs for errors or slow processing warnings
  2. Check consumer CPU/memory — are threads starved?
  3. Profile the processing logic — what's taking time?
  4. Quick win: scale consumers (if under partition count)
  5. If lag is due to one bad message: consider DLT routing

Permanent fix

Optimize processing logic. Increase parallelism (more partitions + consumers). Add async processing where safe. Implement DLT for poison messages. Add monitoring and circuit breakers.

Prevention

Set consumer lag alerts before they become critical. Load test consumers at 3× expected production rate. Implement DLT from day one. Right-size max.poll.records vs processing time.

Reproduce it in the simulator

In Simulator: observe consumer lag in the consumer groups panel. Send multiple messages and watch lag = LEO - consumer offset.

The interactive simulator is being migrated to /simulate.

The interview angle

Growing lag is the #1 consumer health problem. Key question from interviewers: 'How do you determine if lag is a temporary spike or a sustained problem?' Answer: trend analysis — is lag growing, stable, or shrinking?