What is the Outbox Pattern and how does Kafka enable it?

The Outbox Pattern addresses the dual-write problem: a service must update its database and publish an event to Kafka, but a Kafka transaction does not span an external database, and the Kafka docs note that exactly-once delivery to other systems requires cooperation from those systems. The pattern itself is an application design practice, not a Kafka feature. A common approach: 1) The business transaction writes its data and inserts an event row into an outbox table in the same database transaction. 2) A separate relay, such as a change-data-capture source connector running on Kafka Connect or a polling publisher, reads the outbox table and produces the events to Kafka. 3) The relay records its progress, for example by marking rows as processed or tracking its position in the database log. Because the relay can crash after publishing but before recording progress, delivery is at-least-once, so consumers should be idempotent, for example by deduplicating on an event ID. The result is that events appear in Kafka only for committed database changes, and eventually for every one of them, so the two systems are eventually consistent.

Expected key concepts: Outbox Pattern, dual-write, outbox table, same database transaction, Kafka Connect, polling publisher, at-least-once, idempotent, eventually consistent