Skip to content
Kafka Is Fun

ScenarioProduction Scenarios

How do you safely add a new topic and deploy a new consumer without downtime?

Reference answer

Existing producers and consumers are not affected by adding a topic, so no downtime is needed. Strategy: 1) Create the topic first with the partition count and replication factor you choose (kafka-topics.sh --create, or a NewTopic bean with a KafkaAdmin bean in Spring, which adds topics to the broker when the context starts). Avoid relying on auto-creation (auto.create.topics.enable defaults to true), because an auto-created topic gets the broker defaults, num.partitions=1 and default.replication.factor=1. 2) Pick auto.offset.reset to match what the new consumer group should do, because it applies only when the group has no committed offset: latest (the default) starts at new records, earliest reads the log from the start. 3) If the consumer must see every record the producers write, deploy the consumer (or give it its committed offset) before the producers start writing, or use earliest. With latest and producers deployed first, records written before the group's position is established are skipped. 4) After the rollout, watch kafka-consumer-groups.sh --describe until lag reaches 0 and monitor for errors. In Spring, the container property missingTopicsFatal (default false) controls whether the container refuses to start when a configured topic is missing on the broker.

Expected key concepts: kafka-topics.sh, KafkaAdmin, NewTopic, auto.offset.reset, earliest, latest, committed offset, kafka-consumer-groups.sh, missingTopicsFatal, auto.create.topics.enable

How did that feel?

Your own verdict, not the score. It shows up next to this question in the list, so you can come back to the shaky ones.