Recommended partitions
6
Why: 3 consumer groups × 2 for parallelism headroom (6) exceeds the throughput requirement (1 at ~50 MB/s per partition).
You can add partitions later, but never remove them — start with headroom.
Architecture Planner
Describe your workload and constraints; get partition, broker, storage and network recommendations with the reasoning behind each number, plus warnings for configurations that will hurt in production. Formulas: ~50 MB/s per partition, 60% NIC utilisation cap, 75% disk usage cap.
Workload
10,000 msg/s
Average production rate
100500,000
512 bytes
Average payload size
64102,400
3 ×
Peak vs average (3× typical)
110
Retention & replication
7 days
How long to keep messages
190
3
RF=3 tolerates 1 broker failure
15
2
Must be < RF for writes to survive a failure
15
Consumers
3 groups
Each reads the full topic independently
120
Infrastructure
2 TB
Usable storage per broker
0.520
1,000 Mbps
Per-broker NIC bandwidth
10010,000
Recommended brokers
6
Why: Driven by storage: 8,652 GB total at 75% usable disk per broker needs 6 brokers (~1,442 GB each).
Bounds — network: 1, storage: 6, RF: 3, floor: 3.
Total cluster storage
8,652 GB
Why: Daily volume × 7 days retention × RF=3. Alert at 70% disk usage.
Peak network throughput
102.5 MB/s
Why: Producers 14.6 + replication 43.9 + 3 consumer groups 43.9 MB/s.
Reference patterns
High Throughput, Low Durability
Event tracking, telemetry, analytics where occasional loss is acceptable.
acks=1 · linger.ms=20 · batch.size=1MB · compression=lz4 · RF=2
Fits: Clickstream, IoT sensors, ad impression tracking
High Durability, Transactional
Financial transactions, orders, payments where zero data loss is required.
acks=all · enable.idempotence=true · RF=3 · minISR=2 · linger.ms=5
Fits: Payments, order events, inventory changes
Event Sourcing / Audit Log
Complete history required. Log compaction keeps latest state per entity key.
cleanup.policy=compact · RF=3 · minISR=2 · acks=all · retention.ms=-1
Fits: Customer profiles, product catalog, configuration state
Streaming Analytics
Real-time aggregations and windowed computations with Kafka Streams.
RF=3 · processing.guarantee=exactly_once_v2 · num.stream.threads=4
Fits: Real-time dashboards, fraud detection, recommendation engines
CDC / Debezium Outbox
Database change capture with guaranteed event ordering and no dual-write.
log.cleanup.policy=delete · RF=3 · Debezium connector · Avro + Schema Registry
Fits: Microservices event propagation, data sync, audit trail