A production broker runs out of disk space. What happens and how do you recover?

The Apache documentation does not describe a full disk explicitly. What it does document (KIP-112) is how a broker treats a log directory that fails with an I/O error: the directory is marked bad (offline), the broker stops serving the replicas in it, and the broker shuts down only if no good log directories remain. In practice a full disk shows up as write failures of this kind. Effects: followers and leaders on that directory stop serving, so partitions it led need a new leader from the ISR, and partitions whose ISR falls below min.insync.replicas reject producers that use acks=all (only the affected partitions, not the whole cluster); with no eligible replica a partition goes offline. Recovery: 1) Add capacity (a larger volume or additional log directory). 2) Reduce retention on large topics (retention.ms, default 7 days; retention.bytes, default -1 meaning no size limit). 3) Delete topics that are no longer needed. 4) Add brokers and move partitions with kafka-reassign-partitions.sh (new brokers get no partitions automatically), using --throttle to limit the replication traffic. Prevention: monitor disk usage and partition size (kafka.log:type=Log,name=Size) and the offline log directory count, and alert well before the disk is full; the threshold is a local choice.

Expected key concepts: log directory, offline, KIP-112, ISR, min.insync.replicas, retention.bytes, retention.ms, kafka-reassign-partitions.sh, throttle, add brokers