Skip to content
Kafka Is FunProgress

AdvancedInternals Deep Dive

Explain Kafka's exactly-once semantics implementation end-to-end.

Reference answer

EOS requires three layers working together: 1) Idempotent producer (enable.idempotence=true): assigns a PID + per-partition sequence number. Broker detects duplicate ProduceRequests via (PID, partition, sequence). 2) Transactions: producer uses beginTransaction/send/commitTransaction with a transactional.id. The coordinator tracks transaction state in __transaction_state. A Transaction Coordinator assigns epoch to each transactional.id to fence old zombies. 3) Read-committed consumers (isolation.level=read_committed): skip abort markers and don't expose uncommitted records. For Kafka Streams: processing.guarantee=exactly_once_v2 atomically commits (output offsets + consumed offsets) in a single transaction.

Expected key concepts: PID, sequence number, duplicate detection, transactional.id, Transaction Coordinator, __transaction_state, zombie fencing, epoch, read_committed, abort markers, processing.guarantee, exactly_once_v2