What is log compaction and when would you use it?

Log compaction ensures Kafka retains the latest value for each record key, removing older records with the same key (except tombstones). Unlike time/size retention that deletes entire segments, compaction operates within segments to preserve the latest state per key. Use log compaction for: change data capture (CDC) topics, application configuration/settings, user profiles, product catalogs — any topic where the latest value per entity is what matters. Combine with deletion via log.cleanup.policy=compact,delete for both features.

Expected key concepts: latest value, key, tombstones, CDC, change data capture, entity state, log.cleanup.policy, compact, delete, configuration