Skip to content
Kafka Is Fun
ClusterBrokers

KafkaStorageException: a broker's log directory has gone offline (disk error)

Checked against Kafka 4.2 · Oct 2026Community draft, not yet reviewed by a Kafka specialist

What it means

When an IOException occurs while accessing a log directory, the directory is marked offline, and in the 4.2 source it stays offline until the broker is restarted. The broker stops serving the replicas in that directory, and if every log directory has failed, or the metadata log directory fails, the broker shuts itself down. Clients see KAFKA_STORAGE_ERROR ("Disk error when trying to access log file on the disk"). A full disk is one way writes to a log directory fail.

Quick answer

Find out which log directory failed and why, fix it, and restart the broker. If the cause is space, free it by lowering retention on the largest topics so Kafka deletes old segments itself, or add disk, and then look at why it filled.

Likely causes, most common first

  1. A log directory failed or filled up

    How to confirm

    The OfflineLogDirectoryCount metric (normal value 0) and the LogDirectoryOffline metric show it. The broker logs "Stopping serving replicas in dir ... because the log directory has failed", and "Shutdown broker because all log dirs ... have failed" when none is left.

    Fix

    Fix the underlying problem, for example free space or replace the disk, then restart the broker.

  2. Retention keeps more data than the disk can hold

    How to confirm

    retention.ms defaults to 604800000 (seven days) and retention.bytes to -1, which means no size limit. kafka-log-dirs.sh --describe queries log directory usage on a broker, so you can see which topics take the space.

    confirm
    kafka-log-dirs.sh --bootstrap-server localhost:9092 --describe
du -sh /var/lib/kafka/data/* | sort -h | tail

    Fix

    Reduce retention on the largest topic. The log documentation says data is deleted one log segment at a time, by time and size policies, and the broker checks for deletable segments every log.retention.check.interval.ms (300000 by default).

    fix
    kafka-configs.sh --bootstrap-server localhost:9092 --alter --entity-type topics --entity-name orders --add-config retention.ms=86400000

  3. Data is spread unevenly, so one disk fills first

    How to confirm

    The log-dirs output shows one broker or directory far larger than the others.

    Fix

    Move partitions with kafka-reassign-partitions.sh, which the operations documentation describes, and size disks with headroom. A replication throttle limits the bandwidth the move uses.

  4. A compacted topic keeps growing

    How to confirm

    The large topic has cleanup.policy=compact. min.cleanable.dirty.ratio (0.5 by default) bounds the space wasted by duplicates, and delete tombstones are kept for delete.retention.ms (one day by default).

    Fix

    Check that the topic needs compaction, and whether those two settings fit your data.

See it happen

Read how this fails in a real incident

Sources

Every claim above comes from these Apache pages, as of Kafka 4.2. If a page does not say it, this entry does not.

Was this helpful?

Report an issue (opens GitHub in a new tab)

Back to all errors