Skip to content
Kafka Is Fun

ScenarioProduction Scenarios

How do you handle schema evolution when you have 50 consumers reading a Kafka topic?

Reference answer

Using Schema Registry with Avro (these compatibility rules are Confluent Schema Registry's, not part of Apache Kafka): 1) Set the subject's compatibility. BACKWARD (the registry default): a consumer using the new schema can read data written with the previous schema, so upgrade all consumers first, then the producers. FORWARD: data written with the new schema can be read by consumers on the previous schema, so upgrade producers first. FULL is both. With 50 consumers that cannot all upgrade at once, FORWARD or FULL lets versions coexist, and the TRANSITIVE variants check against all earlier versions, not just the previous one. 2) Safe changes: add a field with a default value; removing a field is compatible only if it had a default. In Avro schema resolution a reader field missing from the writer without a default is an error, and writer fields the reader does not know are ignored. Renaming a field needs aliases, otherwise treat it as breaking. 3) Check before registering: the registry has a compatibility check API (POST /compatibility/subjects/<subject>/versions/<version>) that uses the subject's configured level, and it can run in CI. 4) For a breaking change, a common approach is a new topic with a versioned suffix (orders-v2) and migrating consumers gradually. 5) Keep a schema changelog.

Expected key concepts: Schema Registry, BACKWARD, FORWARD, FULL, default value, aliases, upgrade all consumers, compatibility check, breaking change, new topic, TRANSITIVE

How did that feel?

Your own verdict, not the score. It shows up next to this question in the list, so you can come back to the shaky ones.