KRaft — Kafka Without ZooKeeper
How KRaft replaced ZooKeeper, enabling Kafka to manage its own metadata via Raft consensus.
- 1Why ZooKeeper Had to GoBefore KRaft (Kafka 3.3+, default in Kafka 4.0), Kafka depended on Apache ZooKeeper for cluster metadata: broker registr…
- 2Raft Consensus & the Controller QuorumKRaft uses the Raft consensus algorithm. A subset of brokers (3 or 5, configurable) form the controller quorum. One is e…
- 3The Metadata LogIn KRaft, all cluster metadata is stored in a special internal Kafka topic: __cluster_metadata. Every change (broker reg…
- 4Controller Epochs & FencingEach controller election increments the controller epoch — a monotonically increasing version number. Brokers always tra…
- 5KRaft vs ZooKeeper: PerformanceKRaft dramatically improves metadata scalability. ZooKeeper-based Kafka could handle ~200k partitions before controller …
Knowledge check
Q1.What is the role of the controller epoch in KRaft?
Q2.How many controller nodes are needed to tolerate 1 failure?
Q3.Where is cluster metadata stored in KRaft mode?