ScenarioProduction Scenarios
A Kafka producer is experiencing high p99 latency. List 5 potential causes and fixes.
Reference answer
Common causes: 1) Metadata fetch: send() can block for the first record sent to the cluster while the producer fetches metadata; a common approach is to warm it at startup by calling partitionsFor(). 2) Batching: linger.ms (default 5 ms since 4.0) makes the producer wait up to that long before sending, so it adds latency to each record, while very small batches mean many requests; tune linger.ms and batch.size together against your latency target. 3) Buffer full: when buffer.memory is exhausted send() blocks for up to max.block.ms; reduce the send rate, raise buffer.memory, or find why the broker is slow. 4) Broker request handler saturation: RequestHandlerAvgIdlePercent is ideally above 0.3, and a lower value means the handler threads are busy, so add broker capacity or reduce load. 5) Follower lag: with acks=all the leader waits for the in-sync replicas (ISR), so a slow follower raises latency; check replica lag and broker disk and network. 6) GC pauses in the producer JVM stall sends; tune the collector or reduce allocation. 7) Network latency between the producer and the brokers: place them close together; delivery.timeout.ms (default 120 s) and request.timeout.ms (default 30 s) bound how long a send can wait. The broker's request total time metric, split into queue, local, remote and response send time, shows where the time goes.
Expected key concepts: metadata fetch, partitionsFor, linger.ms, buffer.memory, max.block.ms, RequestHandlerAvgIdlePercent, follower lag, acks=all, ISR, GC pauses, network latency
How did that feel?
Your own verdict, not the score. It shows up next to this question in the list, so you can come back to the shaky ones.