Skip to content
Kafka Is FunProgress

MediumData Quality

Message Duplication on Producer Retry

Symptoms

  • Same business events appearing multiple times downstream
  • Database constraint violations on unique keys
  • Order count doubled in analytics

Root cause

Producer receives a network timeout and retries, but the original request was already successfully written by the broker. Without enable.idempotence=true, the broker has no way to detect the duplicate ProduceRequest and appends the message twice. This is at-least-once delivery without idempotency protection.

Detection

Compare message counts to source counts. Check producer error counters for NETWORK_EXCEPTION retries. If not using enable.idempotence=true, duplicates are possible on any network failure.

Immediate actions

  1. Check if the downstream system handles duplicates idempotently (upsert vs insert)
  2. Verify message uniqueness using a message ID field (UUID in headers)
  3. Short-term: deduplicate at consumer using Redis SET with message ID + TTL

Permanent fix

Enable enable.idempotence=true on all critical producers. Add unique message IDs (UUID) to every message. Implement idempotent consumer logic (upsert by message ID).

Prevention

enable.idempotence=true by default. Never use retries without idempotency. Design consumers to be idempotent from the start. Test duplicate handling explicitly.

Reproduce it in the simulator

Watch the producer pipeline: when NOT_LEADER_OR_FOLLOWER occurs, the producer retries. With enable.idempotence=true (simulated), duplicates don't appear.

The interactive simulator is being migrated to /simulate.

The interview angle

Classic interview question: 'How do you prevent Kafka duplicates?' Answer: enable.idempotence=true prevents producer-side duplicates per session. For cross-session: transactions. For consumer-side: idempotent processing logic.