Skip to content
Kafka Is Fun

CriticalResource Exhaustion

Disk Full on Broker

Symptoms

  • IOException: No space left on device in broker logs
  • Broker stops accepting writes (ProduceResponse with error)
  • Replication to this broker stops
  • If this broker has partition leaders: those partitions go offline

Detection

OS disk usage alerts > 80%. Broker logs: java.io.IOException: No space left on device. kafka.log:LogEndOffset stops advancing on all partitions on the full broker.

Challenge mode: diagnose the root cause from the symptoms before reading the answer.

Root cause

The broker's log directory disk has reached capacity. Kafka cannot append to any partition log on that disk. Log retention deletion may be delayed because the active segment is never deleted even if expired.

Immediate actions

  1. IMMEDIATE: Free disk space — delete old segments manually (dangerous), or resize the volume
  2. Reduce retention: kafka-configs.sh --alter --entity-type topics --entity-name <topic> --add-config retention.ms=86400000
  3. Move leadership away from full broker: kafka-preferred-replica-election.sh
  4. Emergency: restart broker after clearing space (triggers log recovery)

Permanent fix

Expand broker disk capacity. Add new log directory and update log.dirs config. Use kafka-reassign-partitions.sh to balance partitions to the new directory.

Prevention

Alert at 70% disk usage. Set retention.bytes per topic to cap partition size. Use multiple log.dirs across different disks. Monitor daily growth rate for capacity planning.

Reproduce it in the simulator

This failure isn't directly simulated, but stopping a broker and observing ISR degradation demonstrates the recovery path.

The interactive simulator is being migrated to /simulate.

The interview angle

Disk full is a silent killer. Retention deletion is segment-granular — the active (current) segment is NEVER deleted even if all its messages have expired. This means you can be surprised by disk usage exceeding expectations.