ScenarioProduction Scenarios
How do you safely increase the partition count of an existing Kafka topic?
Reference answer
Important caveat: increasing partitions BREAKS key-based partitioning for existing messages — the same key may now hash to a different partition. Steps: 1) Verify no key-based ordering requirement exists, OR plan to handle the dual-partition period. 2) Use kafka-topics.sh --alter --partitions N (can only increase, never decrease). 3) Existing messages in old partitions remain there — new messages distribute across all partitions. 4) Consumer group will rebalance — expect brief pause. 5) If you need to maintain key affinity: create a new topic with more partitions, migrate producers to new topic, migrate consumers. 6) Test in staging first — verify consumer behavior after partition count change.
Expected key concepts: partition count, key-based partitioning breaks, kafka-topics.sh --alter, can only increase, consumer rebalance, existing messages, key affinity, new topic, migration, staging