Skip to content
Kafka Is FunProgress

ScenarioProduction Scenarios

A Kafka Streams application is falling behind. How do you scale it?

Reference answer

Scaling Kafka Streams: 1) Add application instances with the same application.id — Kafka Streams automatically rebalances partition assignments across instances. 2) Increase num.stream.threads per instance (processes multiple partitions per JVM). 3) Increase input topic partition count (enables more parallelism). 4) Optimize state store operations (RocksDB tuning, reduce state store reads). 5) Use standby replicas (num.standby.replicas=1) for faster failover. 6) Check for data skew — hot partitions causing one instance to do most work. 7) Tune cache.max.bytes.buffering for more aggressive in-memory caching before flushing.

Expected key concepts: application.id, application instances, rebalance, num.stream.threads, partitions, RocksDB, standby replicas, data skew, hot partitions, cache.max.bytes.buffering, scaling