🔁Message Duplication on Producer Retry

Root cause

After a network error the producer cannot tell whether the broker wrote the record before the connection failed, so it sends it again. Without idempotence a retry can write the record twice. enable.idempotence defaults to true, but it only covers retries within a single producer session; a restarted producer, or an application that resends, is not covered. Consumers that read a record twice (at-least-once) produce the same symptom.

Immediate actions

1. Check whether the downstream system can ignore a repeat (an upsert instead of an insert)
2. Check whether each record carries a unique ID the consumer can use to spot a repeat
3. Make sure enable.idempotence is not turned off

Permanent fix

Keep enable.idempotence=true (the default). For atomic writes across partitions and recovery across producer restarts, use transactions (transactional.id). Make consumers tolerate redelivery.