Architecture Planner

Describe your workload. Get a cluster size, how it behaves when brokers fail, and the settings to copy, with the reasoning behind every number. Formulas: about 50 MB/s per partition, 60% NIC use, 75% disk use.

Start from an example workload

Your cluster

No warnings · storage is the bottleneck

brokers

partitions

8.4 TB

total storage

88 MB/s

peak network

3 partitions cannot put data on all 6 brokers.

What decides the broker count

You need the largest of these. Storage is the bottleneck.

Network2 brokers

Storage6 brokers

Replication factor3 brokers

Minimum3 brokers

Each broker holds about 1.4 TB of its 2 TB disk (70%). The model caps usable disk at 75%.

Biggest levers

Change one thing, see what it does to the cluster.

Retention 7 → 3 days6 → 3 brokers
Messages half the size6 → 3 brokers
For example by compressing or trimming payloads.
Replication factor 3 → 26 → 4 brokers
Survives one fewer broker failure.

If brokers fail

With replication factor 3 and min.insync.replicas 2.

Data survives: 2 failures
acks=all writes continue: 1 failure
Reads continue: 2 failures

Why these numbers

Partitions: 3Show the working

A floor of 3 for basic parallelism; your throughput needs fewer. Consumer groups are independent readers of the whole topic and do not add partitions; only consumer instances inside one group would. You can add partitions later but never remove them, so leave headroom.

Storage: 8.4 TBShow the working

412 GB a day × 7 days of retention × 3 copies. Alert at 70% disk usage.

Peak network: 88 MB/sShow the working

Producers 14.6 + replication 29.3 + 3 consumer groups 43.9 MB/s.

Assumptions and what is left outShow the working

About 50 MB/s per partition, 60% NIC use and 75% disk use. Storage uses average traffic; network uses peak. Not modelled: compression, the number of consumer instances in a group, rack awareness, tiered storage, and spare disk for partition reassignments.

6 brokers · 3 partitions

8.4 TB · 88 MB/s