HighError Handling
Dead Letter Queue Flood
Symptoms
- The dead letter queue topic's offsets climb by thousands of records in a short time
- An alert on the DLQ has been firing for a long while with nobody acting on it
- A sample of DLQ records shows the same exception class again and again
- The connector keeps running, because errors.tolerance=all lets it continue past records that fail
Detection
Read a small, recent sample of the DLQ rather than the whole backlog. If errors.deadletterqueue.context.headers.enable=true (it is optional and off by default), each record has headers such as __connect.errors.exception.class.name, __connect.errors.exception.message and __connect.errors.topic. Tally the exception classes: one class that dominates the sample is one incident, not thousands.
Challenge mode: diagnose the root cause from the symptoms before reading the answer.
Root cause
A Kafka Connect sink connector writes a record to its dead letter queue (errors.deadletterqueue.topic.name) when it fails in the converter or in a transform and errors.tolerance is all. A flood is usually one or two causes repeated for every record: a producer that started writing something the converter cannot read, such as a schema change, or a field a transform expected that is now missing or null. The queue only receives these converter and transform failures: errors raised while the connector writes to the sink system (in put()) are not routed to it.
Immediate actions
- Sample a handful of recent DLQ records, with their headers and payload, before you change anything
- Count them by exception class to separate the cause from its repetition
- Read one representative record per class in full: key, payload, headers and the original topic
- Remember the DLQ covers sink connectors only, and a healthy main topic does not mean the data in the sink is complete
Permanent fix
Fix the cause where it starts: correct the producer's output, adjust the transform, or fix the converter configuration. Raising retries or DLQ capacity does not remove it.
Prevention
A configuration the Kafka Connect documentation shows: errors.tolerance=all, errors.deadletterqueue.topic.name, errors.log.enable=true, with errors.deadletterqueue.context.headers.enable=true added so the records say why they failed. Alerting on how fast the DLQ grows, and having a replay plan before you need one, are common practice rather than something Apache documents.
Reproduce it in the simulator
This failure lives in Kafka Connect, not in the cluster simulator. Work through it in the "Dead Letter Queues & Triage" lesson and the matching Practical Lab.
Open the simulator to try it.
The interview angle
A strong answer samples first, groups by exception class, reads one representative record, and fixes the cause at its source. A weak answer reads records one by one or just raises retries.
Sources
- Kafka Connect user guide (4.2 documentation) (opens in a new tab)
- Kafka Connect configs (4.2 documentation) (opens in a new tab)
- KIP-298: Error Handling in Connect (opens in a new tab)
Checked against Kafka 4.2 documentation on 11 October 2026. Draft: not yet reviewed by a Kafka specialist, and releases move on, so confirm against the current documentation before you act on it.