What is the __transaction_state topic and how does it work?

__transaction_state is an internal topic that serves as the transaction log: a persistent, replicated record of every transaction. It has 50 partitions and replication factor 3 by default (transaction.state.log.num.partitions and transaction.state.log.replication.factor), and it is created with the compact cleanup policy. Each transactional.id maps to one of its partitions via abs(transactionalId.hashCode) % transaction.state.log.num.partitions, and the Transaction Coordinator for that transactional.id is the broker that leads that partition (this follows the 4.2 source; the prose docs do not spell it out). InitProducerId records the transactional.id to producer ID (PID) and epoch mapping in the log and bumps the epoch, which fences older instances. beginTransaction() only changes local producer state; the coordinator logs a partition when the first AddPartitionsToTxn request for it arrives. Each produced batch carries the PID, epoch and sequence number. On commitTransaction() the coordinator writes a PREPARE_COMMIT record to the log, sends WriteTxnMarkers requests so each partition's broker appends a COMMIT control marker, and then writes the final COMMITTED record. Because the log is persistent and replicated, a new coordinator can rebuild state from it and roll incomplete transactions forward or back.

Expected key concepts: __transaction_state, Transaction Coordinator, transactional.id, transaction.state.log.num.partitions, PID, epoch, sequence, AddPartitionsToTxn, PREPARE_COMMIT, compact