Producer Deep Dive
Batching, compression, partitioning strategies, retries, and idempotency — from first principles.
- 1Producer Lifecycle & BatchingWhen you call producer.send(), the record is immediately placed in a RecordAccumulator (an in-memory buffer organized by…
- 2Partitioning StrategiesWithout a key, the producer uses the sticky partitioner (Kafka 2.4+): it sticks to one partition until the batch is full…
- 3Retries & IdempotencyWhen a ProduceRequest fails with a retryable error (LEADER_NOT_AVAILABLE, NOT_LEADER_OR_FOLLOWER), the producer retries …
- 4Producer TransactionsKafka transactions allow atomic writes across multiple partitions and topics. A transactional producer uses a transactio…
- 5Delivery SemanticsAt-most-once: acks=0 or acks=1 without retries. Messages may be lost but never duplicated. At-least-once: acks=all with …
Knowledge check
Q1.What does enable.idempotence=true prevent?
Q2.What does linger.ms control?
Q3.Without a message key, how does Kafka 2.4+ choose a partition?