Kafka Glossary
The 26 terms you need to read anything else on this site — each with a one-line definition and the longer story.
- Broker
- A Kafka server that stores partition data and serves producers and consumers.
- A broker is a single Kafka server process. It hosts partition replicas, accepts writes from producers, serves reads to consumers, and replicates data to and from other brokers. A production cluster runs multiple brokers so partitions can be replicated across machines for fault tolerance.
- Topic
- A named category of events that producers write to and consumers read from.
- A topic is a logical stream of records, identified by name (for example `orders`). Topics are split into partitions for parallelism. Unlike a queue, a topic retains records for a configured period regardless of whether they have been consumed, so multiple independent consumer groups can each read the full stream.
- Partition
- An ordered, append-only log; the unit of parallelism and ordering in Kafka.
- Each topic is divided into partitions. A partition is an ordered sequence of records that events are appended to, stored on disk, and Kafka states its ordering guarantee per partition. Partitions can live on different brokers, which is what lets Kafka scale horizontally, and the partition count caps how many consumers in a group can work in parallel.
- Offset
- The position of a record within a partition.
- Each record in a partition has an offset, a single integer giving its position in that partition. Offsets are per partition, not per topic. Consumers track their progress by offset, and committing an offset records how far a group has got.
- Leader
- The replica that receives all writes for a partition.
- Under normal conditions each partition has a single leader replica. All writes go to the leader. Reads normally go to it too, although followers can serve them (KIP-392), and followers replicate by fetching from the leader. If the leader's broker fails, the controller elects a new leader, from the in-sync replicas when there are any, so the partition stays available.
- Follower
- A replica that copies the leader's log by fetching from it.
- Followers continuously fetch records from the partition leader to keep a copy of its log. By default consumers read from the leader, and followers can serve reads when configured to (KIP-392). A follower that keeps up with the leader is part of the ISR, which is the first choice when a new leader is elected.
- Replica
- One copy of a partition's log, hosted on a broker.
- The replication factor determines how many replicas each partition has. One replica is the leader; the rest are followers. Replicas are placed on different brokers so that losing a machine does not lose data.
- ISR
- In-Sync Replicas: the replicas keeping up with the leader.
- The ISR is the set of replicas the leader considers caught up. A follower is removed if it has not sent fetch requests, or has not consumed up to the leader's log end offset, for `replica.lag.time.max.ms` (30 seconds by default). With `acks=all` the leader waits for the full ISR to acknowledge a write, and `min.insync.replicas` sets how small the ISR may get before writes are refused. The ISR is the first choice when electing a leader. The UnderReplicatedPartitions metric counts partitions whose ISR is smaller than their replica list.
- Consumer group
- A set of consumers that share the partitions of a topic between them.
- Kafka assigns each partition to exactly one consumer within a group at a time, spreading work across members. Separate groups are independent: each gets every record and tracks its own offsets. Group membership changes trigger a rebalance that redistributes partitions.
- Committed offset
- The offset up to which a consumer group has recorded its progress.
- When a consumer commits, the group coordinator stores the offset in the internal `__consumer_offsets` topic. After a restart the consumer resumes from its committed offset. Records processed after the last commit are processed again after a crash, which is at-least-once behaviour.
- Rebalance
- The redistribution of partitions among the consumers in a group.
- A rebalance runs when members join or leave a group, when the coordinator removes one (it sent no heartbeats within the session timeout, or did not call poll() within `max.poll.interval.ms`), or when subscriptions change. With an eager assignor every member revokes its whole assignment first. The cooperative-sticky assignor revokes only the partitions that must move, and Kafka's default assignor list, [RangeAssignor, CooperativeStickyAssignor], is eager until RangeAssignor is removed.
- Idempotence
- Producer setting that makes retries safe: no duplicate copies on resend.
- With `enable.idempotence=true`, the default since Kafka 3.0, the broker assigns each producer an ID and deduplicates messages using a sequence number the producer sends with every message. It prevents duplicates caused by producer retries within a single producer session. Setting `transactional.id` implies it.
- High watermark
- The offset up to which all in-sync replicas have replicated; the limit for what consumers are given.
- Only committed messages, those replicated by all ISR members, are given out to consumers. The leader may hold messages beyond the high watermark (HWM), up to its log end offset, that are not replicated yet. The HWM cannot advance while the ISR is smaller than `min.insync.replicas`, and consumers using `read_committed` are limited further, to the last stable offset.
- Log end offset
- The next offset that will be written to a partition (LEO).
- The LEO marks the tip of a partition's log on a given replica. On the leader, records between the high watermark and the LEO have been written locally but not yet fully replicated, so they are not given out to consumers until the HWM catches up.
- Lag
- How far a consumer group is behind the end of a partition.
- `kafka-consumer-groups.sh --describe` shows CURRENT-OFFSET, LOG-END-OFFSET and LAG for each partition, where LAG is LOG-END-OFFSET minus CURRENT-OFFSET. Lag near zero means the group is caught up, and continuously growing lag means it processes records more slowly than they arrive. Kafka measures lag in records, not in time.
- Controller
- A node that holds the cluster metadata; in KRaft mode, one active controller leads the others.
- In KRaft mode a set of controllers keeps all the cluster's metadata in memory and on disk. One controller is active and the others are hot standbys, and together they form a Raft quorum over the `__cluster_metadata` log. The active controller elects partition leaders. KRaft replaced ZooKeeper, which Kafka 4.0 no longer supports.
- Retention
- How long (or how much) data a topic keeps before old segments are deleted.
- Retention is controlled by `retention.ms` (time, default 7 days) and `retention.bytes` (size per partition, default unlimited). Under the delete policy Kafka deletes whole segments: with time-based retention a segment becomes eligible only when all its records are older than the threshold.
- Compaction
- Cleanup mode that keeps at least the latest record per key.
- With `cleanup.policy=compact` (broker default `log.cleanup.policy`), Kafka retains at least the last known value for every key and removes older values in the background; the head of the log has not been compacted yet. Deletes are expressed as tombstones (null values) that are cleaned out after `delete.retention.ms`. The topic holds a snapshot of the latest value per key, not the complete history.
- Dead Letter Queue (DLQ)
- A topic that catches records a pipeline gave up on processing.
- Kafka Connect sink connectors have built-in dead letter queue settings, and Kafka Streams has `errors.dead.letter.queue.topic.name`; with the plain consumer API you build one yourself. A record that keeps failing is routed to the DLQ so the rest of the stream keeps flowing. A DLQ is only useful if someone is watching it.
- Poison Pill
- A message that fails the same way every time, no matter how many times it's retried.
- A poison pill is a record that fails deterministically, such as a malformed payload or an incompatible schema, so retrying it never succeeds. (The term is general engineering vocabulary, not an Apache Kafka setting.) The usual remedy is to stop retrying after a bounded number of attempts and set the record aside, for example in a dead letter queue.
- Fallacies of Distributed Systems
- Eight classic assumptions that go wrong in distributed systems (general lore, not an Apache document).
- The list: the network is reliable; latency is zero; bandwidth is infinite; the network is secure; topology doesn't change; there is one administrator; transport cost is zero; and the network is homogeneous. Sources credit different people for different items, so none is named here. Apache does not say Kafka's design was derived from the list, but each fallacy points at a problem that Kafka's documentation addresses with a concrete mechanism.
- KRaft
- Production-ready in 3.3
- ZooKeeper mode: removed in 4.0
- Kafka's built-in metadata quorum, which replaced ZooKeeper.
- KRaft (Kafka Raft) keeps cluster metadata in an internal log, __cluster_metadata, that a quorum of controller nodes maintains with the Raft protocol. It was marked production-ready for new clusters in Kafka 3.3 and is the only mode from Kafka 4.0, where ZooKeeper mode was removed. A ZooKeeper cluster has to be migrated to KRaft, on a 3.9 release, before it can be upgraded to 4.0.
- Dual-write
- The ZooKeeper-to-KRaft migration phase in which every broker runs KRaft but the controller still writes metadata to ZooKeeper too.
- During a migration the cluster passes through an initial phase, a hybrid phase (some brokers in ZooKeeper mode, with a KRaft controller) and the dual-write phase, where all brokers are in KRaft mode while the controller keeps ZooKeeper up to date. Rolling back is possible until you finalize the migration, which stops the writes to ZooKeeper.
- KIP-848
- GA in 4.0
- The new consumer group protocol, in which the broker coordinates assignment and only the partitions that change owner pause.
- KIP-848 replaces the group-wide synchronization barrier of the classic rebalance protocol. The group coordinator computes a target assignment with a server-side assignor (uniform or range) and each member reconciles toward it through its heartbeats. Clients opt in with group.protocol=consumer. Client-side assignors are not supported, and the 4.0 documentation notes rack-aware assignment is not fully supported.
- Tiered storage
- Production-ready in 3.9
- Keeping older log segments in remote storage.
- Tiered storage moves older log segments to remote storage. It has been production-ready since Kafka 3.9, which also added the ability to disable it per topic. It is not the same as diskless topics (KIP-1150), a proposal that keeps all data in object storage and is not in any release.