A consumer group suddenly shows 100,000 messages of consumer lag. How do you respond?

Lag is LOG-END-OFFSET minus CURRENT-OFFSET per partition, in records, so first judge it against the produce rate. Triage: 1) Check whether the consumers are running and assigned partitions (kafka-consumer-groups.sh --describe --group <group>, which shows CURRENT-OFFSET, LOG-END-OFFSET and LAG per partition, and --members or --state). 2) Check consumer logs for errors or exceptions. 3) Check consumer CPU, memory and GC. 4) Decide whether it is a processing bottleneck (profiling, a slow downstream call) or a connectivity or rebalance problem. 5) Check recent deployments for a new slow code path. Remediation: 1) Add consumers to the group, which helps only up to the number of partitions. 2) Fix the processing bottleneck. 3) If the lag is already shrinking, just monitor. 4) If the backlog is not worth processing, stop the consumers and reset offsets with kafka-consumer-groups.sh --reset-offsets --to-latest --execute (the group must be inactive; without --execute it only shows the plan), accepting that the skipped records are never processed. 5) A common approach for slow processing is parallel processing inside the consumer. Check the trend: growing lag means consumers are still falling behind, stable lag means they consume as fast as producers write, shrinking lag means they are catching up.

Expected key concepts: kafka-consumer-groups.sh, LOG-END-OFFSET, CURRENT-OFFSET, consumer logs, CPU, GC, processing bottleneck, reset-offsets, to-latest, growing lag