A Kafka Streams application is falling behind. How do you scale it?

Kafka Streams parallelism is bounded by the number of stream tasks, which follows the number of input topic partitions. First find where the time goes (lag per partition, processing and state store latency). Then: 1) Start additional instances of the same application with the same application.id; tasks are redistributed automatically and the state of a moved task is recreated from its changelog topic. 2) Increase num.stream.threads per instance (default 1); threads share nothing, and you can use up to as many threads as there are input partitions. 3) Once every task is running, further instances or threads sit idle (they only take over when another fails), so more parallelism needs more input partitions. 4) Optimise state stores: RocksDB can be tuned through rocksdb.config.setter, and the record cache is sized by statestore.cache.max.bytes (default 10 MB; cache.max.bytes.buffering is deprecated). 5) Standby replicas (num.standby.replicas, default 0) shorten failover and state restoration but do not add throughput. 6) A common check is data skew: if most records fall on a few partitions, those tasks become the bottleneck.

Expected key concepts: application.id, stream tasks, num.stream.threads, input partitions, changelog topic, RocksDB, rocksdb.config.setter, standby replicas, statestore.cache.max.bytes, data skew, instances