What is enable.auto.commit and why is it dangerous?

enable.auto.commit (default true) makes the consumer commit offsets periodically in the background, every auto.commit.interval.ms (default 5000 ms). The commits happen on a timer, not when your processing finishes, which is the risk. Auto-commit gives at-least-once delivery only if you consume all the data returned by poll before the next poll or before closing the consumer; otherwise the committed offset can run ahead of what was actually processed, so after a crash the unprocessed records are skipped and lost. If the consumer crashes after processing but before the next commit, those records are delivered again and are duplicated. With manual control (enable.auto.commit=false) you call commitSync() or commitAsync() once the records have been processed, for example after inserting them into a database, which gives at-least-once delivery. The offset you commit should be the next record to consume.

Expected key concepts: enable.auto.commit, auto.commit.interval.ms, 5000, timer, crash, lost, duplicated, commitSync, commitAsync, at-least-once