BeginnerOperations
How do you increase the throughput of a Kafka consumer?
Reference answer
Common approaches: 1) Add consumers to the group; each partition is consumed by exactly one consumer per group, so consumers beyond the partition count get no work. 2) If every partition is already busy, add partitions to the topic; adding partitions does not change the partitioning of existing data, so it can disturb consumers that rely on key-to-partition mapping, and partitions cannot be reduced. 3) Raise fetch.min.bytes (default 1) so the broker returns larger responses, trading some latency; fetch.max.wait.ms (default 500) bounds how long it waits. 4) Tune max.poll.records (default 500), which only caps how many records one poll() returns and does not affect fetching. 5) Use one consumer per thread, since KafkaConsumer is not thread-safe, or hand records to a pool of processing threads, which makes committing offsets harder because threads must coordinate. 6) Reduce per-record processing time.
Expected key concepts: consumers, partitions, fetch.min.bytes, fetch.max.wait.ms, max.poll.records, thread-safe, one consumer per thread, processing threads, processing time, commit
How did that feel?
Your own verdict, not the score. It shows up next to this question in the list, so you can come back to the shaky ones.