ScenarioProduction Scenarios
How do you migrate a Kafka topic to a new cluster with zero data loss?
Reference answer
Migration outline using MirrorMaker 2 (MM2), which is built on Kafka Connect and replicates topics, their configs and consumer group offsets between clusters. 1) Run MM2 from the source to the destination cluster for the topic and the consumer groups. By default remote topics are renamed with the source cluster alias as a prefix ({source}.{topic}, DefaultReplicationPolicy), so consumers on the destination subscribe to the prefixed name unless you configure another replication.policy.class. 2) Wait until the replication lag is close to zero. 3) Pause the producers writing to the source. 4) Let MM2 drain the remaining records and confirm the destination has caught up. 5) Move the consumers: MM2's checkpoint connector emits translated consumer offsets (emit.checkpoints.enabled, default true), and sync.group.offsets.enabled (default false) writes them into the destination's __consumer_offsets, otherwise you apply the translated offsets yourself. The offsets are translated because they are not guaranteed to match the source offsets. 6) Point the producers at the destination. 7) Verify, then retire MM2 and the source topic. The time between pausing the producers and restarting them on the destination is the cutover window, so keep it short. MM2 also supports exactly-once writes for dedicated MirrorMaker clusters since 3.5.0 (exactly.once.source.support).
Expected key concepts: MirrorMaker 2, MM2, Kafka Connect, replication lag, checkpoint, sync.group.offsets.enabled, replication.policy.class, cutover window
How did that feel?
Your own verdict, not the score. It shows up next to this question in the list, so you can come back to the shaky ones.