What is the Kafka Raft (KRaft) consensus algorithm and how does it differ from ZooKeeper-based metadata management?

KRaft (KIP-500, KIP-595) replaces ZooKeeper with a metadata quorum run by Kafka itself. Nodes with process.roles=controller (or broker,controller in combined mode) form the quorum; the controller nodes elect a leader, the active controller, from among themselves without any external system, and the others are hot standbys. Cluster metadata is kept in a replicated metadata log, and brokers consume metadata events from it, so state can be replayed and a new controller does not need a lengthy reload. The protocol is Raft-based: voters elect a leader in numbered epochs by majority vote, a record is committed once it is replicated to a majority of voters, and followers replicate by sending Fetch requests to the leader instead of the leader pushing. A majority of controllers must be alive: 3 controllers tolerate 1 failure and 5 tolerate 2. ZooKeeper's Zab belongs to the same family of majority-based consensus algorithms, so the main differences are architectural: no external dependency to deploy, metadata stored in a Kafka log, and, as a stated goal of KIP-500, support for more partitions with less CPU. Kafka 4.0 removed ZooKeeper mode entirely.

Expected key concepts: KRaft, Raft, active controller, majority, metadata log, epoch, process.roles, hot standby, ZooKeeper, Zab