Skip to content
Kafka Is FunProgress

CriticalConsumer Failure

Consumer Offset Desync / Message Loss

Symptoms

  • Messages processed less than expected — some seem 'missing'
  • Consumer starts reading from unexpected offset after restart
  • Business metrics show gaps in event processing

Root cause

Consumer commits offsets BEFORE successfully processing the corresponding messages. Causes: enable.auto.commit=true (commits on timer regardless of processing success), committing in a finally block even on exception, or committing offset N before the async processing of message N completes.

Detection

Compare Kafka message count with downstream processed count. Check consumer logs for any processing errors between commits. Audit commit timing relative to processing completion.

Immediate actions

  1. Stop the consumer immediately to prevent further offset advancement
  2. Determine the last offset of successfully PROCESSED (not just consumed) messages
  3. Reset consumer offsets to that position: kafka-consumer-groups.sh --reset-offsets
  4. Fix the commit logic before restarting

Permanent fix

Disable auto-commit. Only call commitSync/commitAsync AFTER verifying successful processing. Use MANUAL or MANUAL_IMMEDIATE ack mode in Spring Kafka.

Prevention

Never use enable.auto.commit=true for critical consumers. Implement idempotent processing so replaying messages is safe. Design for at-least-once with idempotent consumers.

Reproduce it in the simulator

Observe the consumer commit step in the Simulator pipeline (Step 10: Offset Commit). Commit happens AFTER successful consumer fetch and processing.

The interactive simulator is being migrated to /simulate.

The interview angle

This is the most common Kafka data loss bug. The fix: enable.auto.commit=false + commitSync/commitAsync only after successful processing. But the real lesson: design idempotent consumers so even if you accidentally commit early, replaying is safe.