Describe your workload and constraints; get partition, broker, storage and network recommendations with the reasoning behind each number, plus warnings for configurations that will hurt in production. Formulas: ~50 MB/s per partition, 60% NIC utilisation cap, 75% disk usage cap.
Average production rate
Average payload size
Peak vs average (3× typical)
How long to keep messages
RF=3 tolerates 1 broker failure
Must be < RF for writes to survive a failure
Each reads the full topic independently
Usable storage per broker
Per-broker NIC bandwidth
Recommended partitions
3
Why: Minimum of 3 partitions for basic parallelism — your throughput needs less. (Consumer groups are independent full-topic readers and don't add partitions; only consumer instances within one group would.)
You can add partitions later, but never remove them — start with headroom.
Recommended brokers
6
Why: Driven by storage: 8,652 GB total at 75% usable disk per broker needs 6 brokers (~1,442 GB each).
Bounds — network: 2, storage: 6, RF: 3, floor: 3.
Total cluster storage
8,652 GB
Why: Daily volume × 7 days retention × RF=3. Alert at 70% disk usage.
Peak network throughput
87.9 MB/s
Why: Producers 14.6 + replication 29.3 + 3 consumer groups 43.9 MB/s.
The same workload under different replication/retention strategies — every number computed by the same model.
| Scenario | RF / minISR | Retention | Partitions | Brokers | Storage | Network |
|---|---|---|---|---|---|---|
| CurrentYour inputs as configured | 3 / 2 | 7d | 3 | 6 | 8,652 GB | 88 MB/s |
| Cost-optimisedRF=2, minISR=1, retention capped at 3 days — cheaper, still tolerates 1 broker failure for acks=all | 2 / 1 | 3d | 3 | 3 | 2,472 GB | 73 MB/s |
| High availabilityRF=3, minISR=2 — survives one broker failure while still accepting writes | 3 / 2 | 7d | 3 | 6 | 8,652 GB | 88 MB/s |
| High throughputRF=2, minISR=1 — halves replication traffic; pair with acks=1 producers that tolerate loss | 2 / 1 | 7d | 3 | 4 | 5,768 GB | 73 MB/s |
Event tracking, telemetry, analytics where occasional loss is acceptable.
acks=1 · linger.ms=20 · batch.size=1MB · compression=lz4 · RF=2
Fits: Clickstream, IoT sensors, ad impression tracking
Financial transactions, orders, payments where zero data loss is required.
acks=all · enable.idempotence=true · RF=3 · minISR=2 · linger.ms=5
Fits: Payments, order events, inventory changes
Complete history required. Log compaction keeps latest state per entity key.
cleanup.policy=compact · RF=3 · minISR=2 · acks=all · retention.ms=-1
Fits: Customer profiles, product catalog, configuration state
Real-time aggregations and windowed computations with Kafka Streams.
RF=3 · processing.guarantee=exactly_once_v2 · num.stream.threads=4
Fits: Real-time dashboards, fraud detection, recommendation engines
Database change capture with guaranteed event ordering and no dual-write.
log.cleanup.policy=delete · RF=3 · Debezium connector · Avro + Schema Registry
Fits: Microservices event propagation, data sync, audit trail