Overview
Sizing starts from your own numbers: producer throughput, the replication factor (each message is stored on RF brokers, so the cluster writes RF times the producer rate to disk), retention, and your latency needs. Apache's hardware guidance is that disk throughput is generally the performance bottleneck and more disks are better, that in some cases network bandwidth is the bottleneck, and that SSL adds CPU cost. For the page cache, the hardware page suggests memory for about 30 seconds of write throughput. Partition count affects the maximum parallelism of your consumers. The documented limits to plan around: log segments and connections use file descriptors (at least 100000 is recommended), and each log segment uses 2 memory map areas, so around 50000 partitions on one broker can exhaust the default vm.max_map_count and crash the broker with an OutOfMemoryError.
Key configuration
# Disk across the cluster ≈ producer MB/s × retention seconds × replication factor
# Per broker ≈ that figure ÷ number of brokers, plus headroom, if partitions are balanced
# Page cache memory ≈ write throughput × 30 s (Hardware and OS page)Production tip
Check yourself
According to Apache's hardware guidance, what is generally the performance bottleneck?
Sources
- Hardware and OS (4.2 documentation) (opens in a new tab)
- Basic Kafka operations (4.2 documentation) (opens in a new tab)
- Design (4.2 documentation) (opens in a new tab)
- Monitoring (4.2 documentation) (opens in a new tab)
Checked against Kafka 4.2 documentation on Oct 2026. Draft: not yet reviewed by a Kafka specialist. Releases move on, so confirm against the current documentation before you act on a number.