Skip to content
Kafka Is FunProgress

HighData Quality

Schema Incompatibility

Symptoms

  • Deserialization exceptions in consumer logs
  • Consumer processing stops (if no DLT configured)
  • Schema Registry returns 409 Conflict on producer registration
  • Consumer lag building on affected partitions

Root cause

A producer deployed a new schema version that is incompatible with the running consumer's expected schema. Example: producer removed a required field or changed a field type. Without Schema Registry compatibility enforcement, producers can publish data consumers can't deserialize.

Detection

Schema Registry conflict on producer startup (if compatibility mode enforced). Consumer logs: org.apache.kafka.common.errors.SerializationException. Check Schema Registry: GET /subjects/{subject}/versions to see schema history.

Immediate actions

  1. Roll back the producer deployment to the last compatible schema
  2. Update consumer to handle the new schema (if forward-compatible change)
  3. Use ErrorHandlingDeserializer + DLT to stop consumer blocking on bad messages
  4. Check Schema Registry compatibility setting: GET /config/{subject}

Permanent fix

Enforce BACKWARD or FULL compatibility in Schema Registry. Always test schema changes with both old and new consumers before deploying. Use Avro with sensible defaults for all new fields.

Prevention

Schema Registry compatibility=BACKWARD minimum. Test with canary deployments. Never remove fields from shared schemas. Always add new fields with default values.

Reproduce it in the simulator

This failure is at the application layer. Use the Spring Boot Center to see how schema evolution is configured with Avro + Schema Registry.

The interactive simulator is being migrated to /simulate.

The interview angle

Schema evolution is a major operational challenge at scale. Key rule: BACKWARD compatibility means consumers can read old AND new data. Deploy consumers first, then producers when adding fields.