IntermediateInternals
Explain the producer request lifecycle with acks=all in detail.
Reference answer
1) The producer serializes the record and chooses a partition: it hashes the key if there is one; with no key the default partitioner uses a sticky partition that changes after batch.size bytes (round-robin only if RoundRobinPartitioner is configured). 2) The record is appended to a batch in the producer's bounded buffer (the RecordAccumulator, sized by buffer.memory); a batch is sent when it is full (batch.size) or linger.ms has elapsed. 3) The background Sender thread builds ProduceRequests from ready batches and sends them to the partition leader. 4) The leader appends the records to its local log and its log end offset (LEO) advances. 5) The followers in the ISR send FetchRequests, pulling the data like normal consumers and appending it to their own logs; with acks=all, writes are accepted only while the ISR size is at least min.insync.replicas. 6) The high watermark (HWM) advances once every in-sync replica has caught up to the new offset, which completes the waiting produce request. 7) The leader sends the ProduceResponse to the producer. 8) On a retriable error or a request.timeout.ms the producer retries until delivery.timeout.ms expires (retries defaults to 2147483647); with idempotence on, which is the default, retries do not create duplicates.
Expected key concepts: RecordAccumulator, Sender, ProduceRequest, partition leader, LEO, ISR, FetchRequests, high watermark, min.insync.replicas, ProduceResponse, delivery.timeout.ms
How did that feel?
Your own verdict, not the score. It shows up next to this question in the list, so you can come back to the shaky ones.