How would you design a Kafka-based event-driven microservices architecture?

Kafka does not prescribe one architecture, so these are common practices, with the Kafka facts that back them. 1) Model topics around domain events and give each topic a clear owner, the producing service, which controls its schema; a shared schema format and a schema registry are often added but are not part of Apache Kafka. 2) Give each service its own consumer group: every group receives all records of the topics it subscribes to independently, while members of one group split the partitions, so sharing a group between services would split the data between them. 3) Use a record key when per-entity ordering matters, because the producer hashes the key to a partition. 4) Expect at-least-once delivery by default and make consumers idempotent, for example by deduplicating on an event ID carried in a header; the idempotent producer is on by default, and transactions are for consume-transform-produce atomicity within Kafka. 5) Route records that fail processing to a dead-letter topic instead of blocking the partition. 6) Use Kafka Streams for stateful processing and Kafka Connect for integration with external systems. 7) Use the Saga pattern for multi-service workflows, since Kafka transactions do not span other systems' databases. 8) Control access with ACLs and set retention and other configs per topic, which also lets internal and external topics differ.

Expected key concepts: domain event, record key, at-least-once, event ID, dead-letter topic, Kafka Streams, Kafka Connect, Saga, ACLs