How would you diagnose and fix high producer latency (P99 > 500ms)?

Start by locating where the time goes. 1) On the producer, compare record-queue-time-avg (the average time record batches spend in the send buffer; high values point to linger.ms, batching or a full buffer, so also look at buffer-available-bytes) with request-latency-avg (the average request latency, covering network and broker time). 2) Check produce-throttle-time-avg to see whether broker quotas are throttling the client. 3) On the broker, check the request handler idle fraction (kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercent; the docs say ideally above 0.3) and NetworkProcessorAvgIdlePercent. 4) With acks=all, a produce request waits in the produce purgatory until the in-sync replicas have the data, so check the produce purgatory size, UnderReplicatedPartitions and follower lag. 5) Rule out client-side pauses such as GC on the producer JVM, and network latency between producer and broker. A common approach to fixing it is to tune linger.ms and batch.size for the workload, spread load over more partitions or brokers, and fix whichever broker, disk or network component the metrics point to.

Expected key concepts: record-queue-time-avg, request-latency-avg, RequestHandlerAvgIdlePercent, produce purgatory, UnderReplicatedPartitions, produce-throttle-time-avg, GC, linger.ms, batch.size, partitions, network latency