IntermediateInternals
How does Kafka handle leader election when a broker fails?
Reference answer
In current Kafka (KRaft; ZooKeeper mode was removed in 4.0): 1) The active controller, the leader of the metadata quorum, relies on broker heartbeats; a broker whose lease (broker.session.timeout.ms) expires without a heartbeat is fenced and removed from all ISRs. 2) The controller identifies the partitions that the failed broker led. 3) For each one it elects a new leader from the remaining members of the ISR: the current leader if still valid, otherwise the first live ISR replica in replica-list order (with Eligible Leader Replicas, a replica from the ELR can be chosen when the ISR is empty). If no eligible replica exists the partition has no leader, unless unclean.leader.election.enable (default false) allows an out-of-sync replica at the risk of data loss. 4) The controller records the new leader and ISR in the metadata log (a PartitionChangeRecord); brokers keep fetching metadata from the controller and update their state, so no LeaderAndIsrRequest is needed (that request belongs to the legacy ZooKeeper mode). 5) Clients learn the new leader by refreshing their metadata, which any broker can answer. If the failed broker was the active controller, another controller in the quorum takes over through Raft.
Expected key concepts: controller, heartbeat, fenced, ISR, new leader, PartitionChangeRecord, metadata log, KRaft, Raft, unclean.leader.election.enable, Eligible Leader Replicas
How did that feel?
Your own verdict, not the score. It shows up next to this question in the list, so you can come back to the shaky ones.