Kafka Glossary

The 18 terms you need to read anything else on this site — each with a one-line definition and the longer story.

Broker: A Kafka server that stores partition data and serves producers and consumers.; A broker is a single Kafka server process. It hosts partition replicas, accepts writes from producers, serves reads to consumers, and replicates data to and from other brokers. A production cluster runs multiple brokers so partitions can be replicated across machines for fault tolerance.
Topic: A named category of events that producers write to and consumers read from.; A topic is a logical stream of records, identified by name (for example `orders`). Topics are split into partitions for parallelism. Unlike a queue, a topic retains records for a configured period regardless of whether they have been consumed, so multiple independent consumer groups can each read the full stream.
Partition: An ordered, append-only log; the unit of parallelism and ordering in Kafka.; Each topic is divided into partitions. A partition is an ordered, immutable sequence of records stored on disk, and ordering is only guaranteed within a single partition. Partitions can live on different brokers, which is what lets Kafka scale horizontally — and the partition count caps how many consumers in a group can work in parallel.
Offset: The monotonically increasing position of a record within a partition.; Every record appended to a partition receives the next integer offset, starting at 0. Offsets are unique per partition, not per topic. Consumers track their progress by offset, and committing an offset records that everything up to that point has been processed.
Leader: The replica that handles all reads and writes for a partition.; Each partition has exactly one leader replica at a time. Producers send writes to the leader; followers replicate from it. If the leader's broker fails, the controller elects a new leader from the in-sync replicas so the partition stays available.
Follower: A replica that copies the leader's log and stands by to take over.; Followers continuously fetch records from the partition leader to keep an identical copy of the log. They serve no client traffic in classic Kafka. A follower that keeps up with the leader is part of the ISR and is eligible to become leader if the current leader fails.
Replica: One copy of a partition's log, hosted on a broker.; The replication factor determines how many replicas each partition has. One replica is the leader; the rest are followers. Replicas are placed on different brokers so that losing a machine does not lose data.
ISR: In-Sync Replicas — the replicas fully caught up with the leader.; The ISR is the set of replicas (including the leader) that have fetched the leader's log within `replica.lag.time.max.ms` (default 30 s). Only ISR members are eligible for clean leader election, and with `acks=all` a write is only acknowledged once every ISR member has it. A shrinking ISR is an early warning of replication problems.
Consumer group: A set of consumers that share the partitions of a topic between them.; Kafka assigns each partition to exactly one consumer within a group, spreading work across members. Separate groups are independent — each gets every record and tracks its own offsets. Group membership changes trigger a rebalance that redistributes partitions.
Committed offset: The last offset a consumer group has durably acknowledged processing.; When a consumer commits, the offset is written to the internal `__consumer_offsets` topic. On restart or rebalance, the group resumes from committed offsets. The gap between a consumer's current position and its committed offset is work that would be repeated after a crash (at-least-once delivery).
Rebalance: The redistribution of partitions among the consumers in a group.; A rebalance runs when consumers join or leave a group, when one is considered dead (missed heartbeats or exceeded `max.poll.interval.ms`), or when subscriptions change. During an eager rebalance all processing pauses; the cooperative-sticky protocol moves only the partitions that must move.
Idempotence: Producer mode that makes retries safe — no duplicates on resend.; With `enable.idempotence=true` the producer gets a producer ID, and every record carries a per-partition sequence number. Brokers discard duplicates caused by retries, giving exactly-once delivery per producer session per partition — the foundation for Kafka transactions.
High watermark: The highest offset replicated to every in-sync replica; the consumer visibility limit.; The high watermark (HWM) advances once all ISR members have a record. Consumers can only read up to the HWM, which prevents them from seeing records that could be lost if the leader failed before replication finished.
Log end offset: The next offset that will be written to a partition (LEO).; The LEO marks the tip of a partition's log on a given replica. On the leader, records between the high watermark and the LEO have been written locally but not yet fully replicated — they are invisible to consumers until the HWM catches up.
Lag: How far a consumer is behind the end of a partition: LEO minus consumer position.; Consumer lag is the number of records between the log end offset and the consumer's position. Steady lag near zero is healthy; continuously growing lag means the consumer processes slower than producers write, and is the single most important consumer health metric.
Controller: The broker that manages cluster metadata and leader elections.; Exactly one active controller manages broker registration, topic creation, partition assignments and leader elections. In KRaft mode the controller role is backed by a Raft quorum over the `__cluster_metadata` log, replacing ZooKeeper and cutting failover to under a second.
Retention: How long (or how much) data a topic keeps before old segments are deleted.; Retention is controlled by `retention.ms` (time, default 7 days) and `retention.bytes` (size, default unlimited). Kafka deletes whole segments, never individual records — a segment becomes eligible only when all its records have expired.
Compaction: Retention mode that keeps only the latest record per key.; With `log.cleanup.policy=compact`, Kafka retains the most recent value for every key and removes older versions, turning the topic into a durable changelog / key-value store. Deletes are expressed as tombstones (null values) that are themselves cleaned up after `delete.retention.ms`.