What is the difference between Kafka's request queue and the Purgatory?

The request queue is where incoming requests wait until a request handler thread is free to process them. Its size is exposed as RequestQueueSize, and RequestHandlerAvgIdlePercent (the average fraction of time the handler threads are idle, ideally above 0.3 according to the Kafka docs) shows how busy those threads are. The Purgatory is a separate structure for delayed operations: requests that cannot be answered yet because they are waiting on a condition. A delayed produce operation waits for the required acknowledgements (with acks=all, until the in-sync replicas have the data), and a delayed fetch waits for enough bytes to accumulate (fetch.min.bytes) or for the fetch wait time (fetch.max.wait.ms) to pass. Each delayed operation has a maximum delay: it completes when its condition is met, or it expires if it is not completed in time. The Purgatory tracks these timeouts with a hierarchical timing wheel. The PurgatorySize metrics (Produce and Fetch) show how many requests are waiting; the Produce one is non-zero when acks=-1 (all) is used.

Expected key concepts: request queue, Purgatory, timing wheel, RequestHandlerAvgIdlePercent, acks=all, fetch.min.bytes, fetch.max.wait.ms, PurgatorySize, delayed