Failure Lab
11 production failure modes, written the way incidents actually unfold: the symptoms you would see, how to detect them, what to do immediately, the permanent fix and how to prevent a recurrence.
HighBroker Failure
Leader Broker Crash
The broker that led one or more partitions stopped: a process crash, a hardware failure or a kill. Until the controller elects a new leader from each partition's in-sync replicas, requests for those partitions fail.
CriticalNetwork Failure
Network Partition
A network partition cuts some brokers off from the others, or from the controller, while their processes keep running. Brokers keep a session with the controller by sending heartbeats; a broker that sends none before broker.session.timeout.ms (9000 ms by default) is considered offline. The controller fences it, removes it from the ISRs, and elects new leaders from the remaining ISR members. Kafka's documentation says the cluster stays available through node failures after a short fail-over, but may not stay available through network partitions.
HighReplication Degradation
ISR Shrink / Under-Replicated Partitions
A follower has not sent fetch requests, or has not caught up to the leader's log end offset, for replica.lag.time.max.ms (30 seconds by default), so the leader removes it from the ISR. Anything that keeps a follower from fetching or keeping up, for example a stalled broker, a slow disk or a congested network, can cause it. The partition keeps working but tolerates fewer failures.
CriticalResource Exhaustion
Disk Full on Broker
The broker's log directory can no longer be written, for example because its disk is full, so the partitions stored there cannot be appended to. The broker reports a failed log directory to the controller; if it cannot do so within log.dir.failure.timeout.ms (30 seconds by default) the broker shuts down.
HighConsumer Failure
Consumer Lag Explosion
The group is reading more slowly than records are being written. Typical causes: slow processing between polls, too few consumers for the partitions, repeated errors on the same records, or one partition that is much busier than the others. Each partition goes to exactly one consumer in a group, so the partition count limits how many consumers can share the work.
HighConsumer Group
Rebalance Storm
A rebalance runs when a member joins, leaves or fails. A member is also removed if it sends no heartbeat within session.timeout.ms (45000 ms by default, classic protocol) or does not call poll() within max.poll.interval.ms (300000 ms by default); it then leaves the group so another consumer can take over. A storm is those events repeating: processing that takes longer than the poll interval, consumers that keep restarting, or pauses long enough to miss heartbeats.
MediumData Quality
Message Duplication on Producer Retry
After a network error the producer cannot tell whether the broker wrote the record before the connection failed, so it sends it again. Without idempotence a retry can write the record twice. enable.idempotence defaults to true, but it only covers retries within a single producer session; a restarted producer, or an application that resends, is not covered. Consumers that read a record twice (at-least-once) produce the same symptom.
CriticalCluster Metadata
Controller Election Loop
The metadata quorum keeps electing new leaders. A controller that gets no fetch response from the quorum leader within controller.quorum.fetch.timeout.ms (2000 ms by default) starts an election by bumping the leader epoch and voting for itself, and a leader that cannot reach a majority of the quorum resigns. A controller that is slow or overloaded, or on an unstable network, can therefore trigger a run of elections.
HighData Quality
Schema Incompatibility
A producer started writing a new schema version that readers cannot handle. Schema Registry compares each new version against the previous one using the subject's compatibility type (BACKWARD by default) and refuses an incompatible one, but a producer that bypasses the check, or a compatibility type that does not match the order in which you upgrade, can still leave consumers unable to read what is on the topic.
CriticalConsumer Failure
Consumer Offset Desync / Message Loss
The consumer's committed offset moved past records it had not finished processing. Saving the position before processing gives at-most-once delivery: after a crash the unprocessed records are not read again. With enable.auto.commit=true the offset is committed in the background, and if the application does not process everything poll() returned before the next poll, the committed offset can get ahead of what was consumed and records are missed. Committing in a finally block, or before asynchronous work completes, has the same effect.
HighError Handling
Dead Letter Queue Flood
A Kafka Connect sink connector writes a record to its dead letter queue (errors.deadletterqueue.topic.name) when it fails in the converter or in a transform and errors.tolerance is all. A flood is usually one or two causes repeated for every record: a producer that started writing something the converter cannot read, such as a schema change, or a field a transform expected that is now missing or null. The queue only receives these converter and transform failures: errors raised while the connector writes to the sink system (in put()) are not routed to it.