What is the purpose of the __consumer_offsets topic?

__consumer_offsets is an internal Kafka topic in which the group coordinator persists consumer group state, including committed offsets. When a consumer commits with commitSync(), commitAsync() or auto-commit, the committed offset is written as a record whose key identifies the group, topic and partition; the committed position is the last offset stored securely, and after a restart the consumer recovers to it. The topic is replicated and compacted (cleanup.policy=compact; offsets.topic.replication.factor defaults to 3 and offsets.topic.num.partitions to 50), so older commits for the same key are eventually removed and the latest one is kept: it holds the current position, not a full history of offsets. How long committed offsets are kept is governed by offsets.retention.minutes (default 10080, i.e. 7 days).

Expected key concepts: __consumer_offsets, internal, committed offset, commitSync, commitAsync, group coordinator, group, topic, partition, compacted, replicated