You're seeing duplicate messages in your consumer application. How do you identify the source and fix it?

Kafka guarantees at-least-once delivery by default, so duplicates must be expected. Sources: 1) Producer retries after a lost acknowledgement. The idempotent producer (enable.idempotence=true, the default in current versions) makes the broker deduplicate by producer ID and sequence number, but only within a single producer session; a restarted producer gets a new ID unless a transactional.id is used. 2) Consumer processes a record and then crashes or loses its partitions (rebalance, or max.poll.interval.ms exceeded) before the offset is committed, so the record is redelivered. This is normal at-least-once behaviour. Note that committing before processing does the opposite: it risks losing records (at-most-once), not duplicating them. 3) Offsets were reset to an earlier position, so check whether anyone ran kafka-consumer-groups.sh --reset-offsets or changed the group id. 4) The same event was produced twice by the application itself. Fixes: make the consumer idempotent, for example attach a unique event ID to each message and record processed IDs in a deduplication store (a database with a unique key or an upsert, or a cache); commit offsets after processing; and enable idempotence or transactions on the producer side.

Expected key concepts: enable.idempotence, producer retries, at-least-once, rebalance, deduplication store, upsert, transactional.id