AdvancedPerformance
How would you handle a Kafka cluster with millions of partitions?
Reference answer
ZooKeeper-based Kafka cannot handle millions of partitions (controller failover takes too long). Use KRaft mode which can handle millions. Additional strategies: 1) Monitor controller CPU — metadata operations scale with partition count. 2) Use dedicated controller nodes (3 or 5) separate from brokers. 3) Increase metadata.log.segment.bytes for larger metadata logs. 4) Tune num.partitions down for unused topics — unused partitions still have overhead. 5) Use topic compaction for key-value topics to prevent unbounded growth. 6) Implement topic lifecycle management (auto-delete inactive topics). KRaft scales linearly while ZooKeeper hits walls at ~200k.
Expected key concepts: KRaft, millions, ZooKeeper, dedicated controllers, metadata.log.segment.bytes, overhead, compaction, topic lifecycle, auto-delete, linear scaling