How would you implement an audit log system using Kafka?

Design: 1) Dedicated audit topic, for example audit.events, keyed by entityId so events for one entity go to the same partition. 2) Do not use log compaction for the trail: compaction keeps at least the last value per key and removes older records for the same key, which would erase audit history. Use the delete cleanup policy with long retention (retention.ms=-1 removes the time limit; retention.bytes defaults to -1, no size limit) and plan disk or tiered storage for the growth. 3) Durability: replication factor 3 with min.insync.replicas=2, producers using acks=all (the default) with enable.idempotence=true, and unclean.leader.election.enable left at its default of false; a committed message is not lost as long as at least one in-sync replica stays alive. Apache gives no recommended replication factor, so higher values are a choice, not a rule. 4) Producers: wait for the acknowledgement (or handle failure in the callback) before treating the audited action as recorded, and retry within delivery.timeout.ms. 5) Separate consumer groups per consumer (compliance, analytics, SIEM) so each reads every event independently. 6) Protect the topic with ACLs: grant Write only to the audit producers, Read to the consumers' groups, and do not grant Delete or Alter. 7) Use a schema format such as Avro and evolve it by adding fields rather than changing existing ones.

Expected key concepts: audit topic, replication factor, min.insync.replicas, retention.ms=-1, compaction, acks=all, enable.idempotence, ACLs, consumer groups, SIEM, Avro