Failure Lab
10 production failure modes, written the way incidents actually unfold: the symptoms you would see, how to detect them, what to do immediately, the permanent fix and how to prevent a recurrence.
HighBroker Failure
Leader Broker Crash
The partition's leader broker process terminated (OOM, hardware failure, OS crash, kernel panic, or manual kill). All produce/consume requests for that partition fail until a new leader is elected from the ISR.
CriticalNetwork Failure
Network Partition (Split-Brain)
A network partition severs communication between some brokers while all processes remain alive. The isolated broker believes it's still the leader; the rest of the cluster elects a new leader. This creates a split-brain window where two brokers may briefly accept writes for the same partition.
HighReplication Degradation
ISR Shrink / Under-Replicated Partitions
One or more follower replicas can no longer keep up with the leader within replica.lag.time.max.ms. Common causes: broker GC pauses, slow disk I/O, network congestion between brokers, high CPU from other workloads. The broker is removed from ISR, reducing fault tolerance.
CriticalResource Exhaustion
Disk Full on Broker
The broker's log directory disk has reached capacity. Kafka cannot append to any partition log on that disk. Log retention deletion may be delayed because the active segment is never deleted even if expired.
HighConsumer Failure
Consumer Lag Explosion
Consumer group is processing slower than messages are being produced. Could be: processing logic too slow (CPU, DB call, API call), too few consumers, an exception causing repeated processing attempts, or a single slow partition causing head-of-line blocking.
HighConsumer Group
Rebalance Storm
Consumers are repeatedly triggering rebalances faster than they can complete. Common causes: max.poll.interval.ms set too low (consumer spends too long processing and gets kicked out, triggering rebalance), rapid pod restarts in Kubernetes, or session.timeout.ms too short relative to GC pauses.
MediumData Quality
Message Duplication on Producer Retry
Producer receives a network timeout and retries, but the original request was already successfully written by the broker. Without enable.idempotence=true, the broker has no way to detect the duplicate ProduceRequest and appends the message twice. This is at-least-once delivery without idempotency protection.
CriticalCluster Metadata
Controller Election Loop
The controller keeps failing and being re-elected. Can be caused by: controller broker GC pauses causing ZooKeeper session timeout, metadata corruption, or network issues specific to the controller broker. Each election increments the epoch and triggers a metadata reconciliation that can overwhelm brokers.
HighData Quality
Schema Incompatibility
A producer deployed a new schema version that is incompatible with the running consumer's expected schema. Example: producer removed a required field or changed a field type. Without Schema Registry compatibility enforcement, producers can publish data consumers can't deserialize.
CriticalConsumer Failure
Consumer Offset Desync / Message Loss
Consumer commits offsets BEFORE successfully processing the corresponding messages. Causes: enable.auto.commit=true (commits on timer regardless of processing success), committing in a finally block even on exception, or committing offset N before the async processing of message N completes.