Skip to content
Kafka Is FunProgress

IntermediateInternals

What is the difference between Kafka's request queue and the Purgatory?

Reference answer

The request queue holds incoming ProduceRequests waiting for handler threads to process them. RequestHandlerAvgIdlePercent < 30% indicates the queue is saturated. The Purgatory (Kafka's deferred processing subsystem using a TimingWheel) holds requests waiting for remote conditions: acks=all ProduceRequests wait here until all ISR replicas fetch the data, and FetchRequests with fetch.min.bytes wait here until enough data accumulates. If a request doesn't complete within request.timeout.ms, it's expired from Purgatory and an error is returned.

Expected key concepts: request queue, Purgatory, TimingWheel, ProduceRequests, ISR replicas, FetchRequests, fetch.min.bytes, request.timeout.ms, handler threads