AdvancedPerformance
How does Kafka handle backpressure from slow consumers?
Reference answer
Brokers do not slow producers because consumers lag: the log decouples them, and the design docs describe a pull model in which a slow consumer simply falls behind and catches up when it can. Backpressure is therefore applied at the edges. 1) Producer: buffer.memory (default 32 MiB) bounds the memory for records waiting to be sent; if records arrive faster than they can be delivered, send() blocks for up to max.block.ms (default 60000 ms) and then throws an exception. 2) Consumer: it pulls at its own pace; max.poll.records (default 500) caps the records returned per poll(), and pause() and resume() suspend fetching from chosen partitions without causing a rebalance (the consumer must still call poll() within max.poll.interval.ms). 3) Kafka Streams: each task processes records one at a time from its per-partition record buffers. 4) Quotas: brokers can enforce byte-rate and request quotas per user and/or client-id, for example producer_byte_rate and consumer_byte_rate set with kafka-configs.sh. A client that exceeds its quota has its channel muted by the broker until the computed delay is over, and it sees this in throttle-time metrics. By default clients have an unlimited quota.
Expected key concepts: backpressure, pull, buffer.memory, max.block.ms, max.poll.records, pause, resume, quota, producer_byte_rate, throttle
How did that feel?
Your own verdict, not the score. It shows up next to this question in the list, so you can come back to the shaky ones.