SpringSpring Kafka Advanced
How do you configure consumer listener concurrency for optimal throughput?
Reference answer
Set ConcurrentKafkaListenerContainerFactory.setConcurrency(N), spring.kafka.listener.concurrency=N, or the concurrency attribute on @KafkaListener (which overrides the factory). The default is 1. A ConcurrentMessageListenerContainer delegates to N child KafkaMessageListenerContainer instances (container.setConcurrency(3) creates three), and each child receives its messages on a single thread with its own KafkaConsumer, appended to client.id as -n. Kafka assigns each partition to exactly one consumer in the group, so concurrency above the number of partitions your instances share leaves some consumers without partitions: for example with three topics of five partitions and concurrency=15 you see only five active consumers (a RoundRobinAssignor can help in that multi-topic case). A common approach is to start from the partition count divided by the number of application instances, for example 12 partitions over 3 instances gives 4, then measure with your own workload, because more threads help only if processing, not the partitions, is the bottleneck. Ordering is still guaranteed only per partition.
Expected key concepts: setConcurrency, spring.kafka.listener.concurrency, ConcurrentMessageListenerContainer, KafkaMessageListenerContainer, partition count, instances, KafkaConsumer, single thread, RoundRobinAssignor, concurrency
How did that feel?
Your own verdict, not the score. It shows up next to this question in the list, so you can come back to the shaky ones.