Skip to content
Kafka Is Fun

HighError Handling

Dead Letter Queue Flood

Symptoms

  • The DLQ topic's offset climbs by thousands of messages in a short window
  • DLQ alerting has been firing for hours (or days) with nobody acting on it
  • A quick sample of DLQ records shows the same exception class over and over
  • The original topic's consumer lag looks completely healthy — the failures aren't blocking the main pipeline

Detection

Don't read DLQ messages one at a time. Pull a small sample (10-20 recent records with headers), tally how often each exception class appears, and treat a DLQ dominated by one exception class as a single incident, not thousands of them.

Challenge mode: diagnose the root cause from the symptoms before reading the answer.

Root cause

A DLQ flood is almost never thousands of unrelated problems — it's usually one or two root causes (a bad producer schema deploy, a rotated downstream credential, a config typo) amplified by batch processing: one failing record can drag every other record in the same batch into the DLQ with it, turning a single bug into a wall of messages.

Immediate actions

  1. Sample a handful of recent DLQ records (with full headers and payload) before touching anything else
  2. Group and count by exception class to separate the root cause from its amplification
  3. Inspect one representative message per exception class in full, rather than skimming the whole batch
  4. Confirm the main topic's consumers are still healthy — a DLQ flood is a different failure mode than a broker or partition outage and shouldn't be responded to the same way

Permanent fix

Fix the failure at its source — correct the producer's schema, add a transform to normalize the offending field, or repair the downstream connector's config or credentials — rather than only raising retry counts or DLQ capacity.

Prevention

Stand up a minimum-viable DLQ configuration from day one (error tolerance on, context headers on, error logging on), alert on DLQ growth rate and message age rather than just message presence, and agree on a replay procedure — retry-count cap, kill switch, side-effect check — before you ever need one under pressure.

Reproduce it in the simulator

This failure lives at the application/connector layer rather than the broker layer, so it isn't directly wired into the cluster simulator. Work through it hands-on via the 'Dead Letter Queues & Triage' lesson and the matching Practice lab.

The interactive simulator is being migrated to /simulate.

The interview angle

Strong interview answer: sample first, group by exception class, diagnose one representative message, then fix at the source. Weak answer: reading through DLQ messages one at a time or just cranking up retries. Also worth mentioning: batch amplification is why a handful of root causes can look like tens of thousands of failures.