Design a high-throughput event ingestion pipeline with Kafka.

Architecture: 1) Producers: batch and compress. Batching is controlled by linger.ms (default 5 since 4.0) and batch.size (default 16384 bytes); a common approach is to raise batch.size and use compression such as lz4 or zstd, because Kafka compresses whole batches and keeps them compressed in the log. Choose acks per topic: acks=all (the default) with min.insync.replicas for data that must not be lost, acks=1 only where losing a record on a leader failure is acceptable. 2) Topics: choose the partition count from measured per-partition throughput and the consumer parallelism you need (consumers in a group are limited by the partition count); a replication factor of 3 is a common choice. 3) Brokers: disk throughput is usually the bottleneck, so use more and faster disks, leave plenty of memory for the OS page cache (caught-up consumers are served from it), and size the network for peak traffic; the example JVM settings in the Apache docs use a 6 GB heap (-Xmx6g). 4) Consumers: process records in batches (tune max.poll.records, default 500) and add consumers up to the partition count. 5) Back-pressure: size buffer.memory (default 32 MB) for bursts; when it is full send() blocks for up to max.block.ms. 6) Monitoring: broker JMX metrics, consumer lag, and RequestHandlerAvgIdlePercent. Load-test with your own message sizes to find the real capacity; Apache documents no universal throughput figure.

Expected key concepts: linger.ms, batch.size, compression, acks, partition count, replication factor, page cache, heap, max.poll.records, buffer.memory, consumer lag