Skip to content
Kafka Is Fun

AdvancedInternals Deep Dive

Explain the Kafka consumer group coordinator and rebalance protocol in detail.

Reference answer

For each consumer group, one broker acts as the Group Coordinator. The group id is hashed (abs(groupId.hashCode) % offsets.topic.num.partitions, 50 partitions by default) to a partition of __consumer_offsets, and the coordinator is the broker that leads that partition (this follows the 4.2 source; the prose docs do not spell it out). Classic protocol: 1) JoinGroup: members find the coordinator and send a JoinGroupRequest with their metadata; once the members have joined, the coordinator selects one as group leader and answers all of them, giving the member list only to the leader. 2) SyncGroup: the group leader, a client, runs the partition assignor and sends the assignments in its SyncGroupRequest (followers send an empty one); every member receives its own assignment in the response. 3) Heartbeat: members keep sending heartbeats to show liveness. The classic protocol relies on a group-wide synchronization barrier. With cooperative rebalancing (CooperativeStickyAssignor, KIP-429), a rebalance may be followed by a second one: partitions that must move are first revoked by their previous owner, which rejoins, and the next rebalance assigns them to their new owners. With group.protocol=consumer (KIP-848, GA in Kafka 4.0), JoinGroup and SyncGroup are replaced by ConsumerGroupHeartbeat, assignments are computed on the coordinator side, and there is no global synchronization barrier.

Expected key concepts: Group Coordinator, __consumer_offsets, JoinGroup, SyncGroup, group leader, assignor, Heartbeat, cooperative rebalancing, revoke, KIP-848, ConsumerGroupHeartbeat

How did that feel?

Your own verdict, not the score. It shows up next to this question in the list, so you can come back to the shaky ones.