ScenarioProduction Scenarios
What would you check first if consumers are reading the same messages multiple times after a deployment?
Reference answer
Checklist: 1) Was the consumer group ID changed? A new group has no committed offsets, so it starts from auto.offset.reset (default latest; earliest would re-read the whole log). 2) Were offsets reset during the deployment, for example a script running kafka-consumer-groups.sh --reset-offsets? 3) Commit timing: with enable.auto.commit (default true) offsets are committed periodically in the background, and with manual commits a bug or a commit after a restart means records processed since the last commit are redelivered. 4) Rebalances during a rolling deployment: when partitions move, records processed but not yet committed are redelivered; exceeding max.poll.interval.ms (default 300000) also triggers one. 5) Was the topic recreated? Offsets committed for the old topic do not apply to a new one. 6) Did the group sit empty longer than offsets.retention.minutes (default 10080, 7 days)? Its committed offsets are then removed and it restarts from auto.offset.reset. 7) Describe the group with kafka-consumer-groups.sh --describe to compare CURRENT-OFFSET and LOG-END-OFFSET. In practice, a group ID change or an offset reset in a deployment script are the first things to rule out.
Expected key concepts: group ID changed, auto.offset.reset, reset-offsets, enable.auto.commit, committed offsets, rebalance, max.poll.interval.ms, offsets.retention.minutes, topic recreated, kafka-consumer-groups.sh, deployment script
How did that feel?
Your own verdict, not the score. It shows up next to this question in the list, so you can come back to the shaky ones.