CriticalResource Exhaustion
Disk Full on Broker
Symptoms
- kafka.log:type=LogManager,name=OfflineLogDirectoryCount is above 0
- kafka.server:type=ReplicaManager,name=OfflineReplicaCount is above 0
- Partitions stored on the failed log directory cannot be written
Detection
OfflineLogDirectoryCount and OfflineReplicaCount, both normally 0. Operating-system disk usage. Apache gives no disk-usage threshold, so choose one for your capacity planning.
Challenge mode: diagnose the root cause from the symptoms before reading the answer.
Root cause
The broker's log directory can no longer be written, for example because its disk is full, so the partitions stored there cannot be appended to. The broker reports a failed log directory to the controller; if it cannot do so within log.dir.failure.timeout.ms (30 seconds by default) the broker shuts down.
Immediate actions
- Free space or grow the volume
- Lower retention on large topics: kafka-configs.sh --bootstrap-server localhost:9092 --alter --entity-type topics --entity-name <topic> --add-config retention.ms=<milliseconds>
- Move replicas off the full broker with kafka-reassign-partitions.sh
- Once there is space, make sure the broker is running again with a usable log directory (a restart if it shut down)
Permanent fix
Add disk capacity or brokers. log.dirs may list several directories: partitions are spread round-robin across them and each partition lives entirely in one. log.dirs is read-only, so adding a directory means changing the configuration and restarting the broker.
Prevention
Cap partition size with retention.ms and retention.bytes, use more than one log directory where it makes sense, and alert on OfflineLogDirectoryCount above 0.
Reproduce it in the simulator
This failure is not simulated. Stopping a broker and watching the ISR shrink shows the recovery path.
Open the simulator to try it.
The interview angle
Retention deletes data one log segment at a time, and time-based retention looks at the largest timestamp in a segment, so disk use can stay above what retention.ms alone suggests.
Sources
- Broker configs (4.2 documentation) (opens in a new tab)
- Topic configs (4.2 documentation) (opens in a new tab)
- Monitoring (4.2 documentation) (opens in a new tab)
- Basic Kafka operations (4.2 documentation) (opens in a new tab)
- Hardware and OS (4.2 documentation) (opens in a new tab)
- Log implementation (4.2 documentation) (opens in a new tab)
Checked against Kafka 4.2 documentation on 11 October 2026. Draft: not yet reviewed by a Kafka specialist, and releases move on, so confirm against the current documentation before you act on it.