Skip to content
Kafka Is Fun

Production Architecture

Sizing, monitoring, security, operations — running Kafka reliably at scale.

  1. 1Cluster SizingThe key inputs for cluster sizing: throughput (MB/s producer), replication factor (× RF for replica traffic), retention …
  2. 2Essential Monitoring MetricsCritical JMX metrics every Kafka team must monitor: • UnderReplicatedPartitions > 0 → replication problem • ActiveContro…
  3. 3Security: Authentication & AuthorizationKafka supports SASL (PLAIN, SCRAM-SHA-256, GSSAPI/Kerberos, OAUTHBEARER) for authentication and TLS for encryption in tr…
  4. 4Topic Design Best PracticesName topics descriptively: {domain}.{entity}.{event-type} e.g. orders.order.created. Avoid broad topics with mixed schem…
  5. 5Operational Runbook EssentialsKey operational procedures: Rolling restart (upgrade brokers one at a time, wait for ISR to recover between each). Prefe…
  6. 6Dead Letter Queues & TriageA dead letter queue (DLQ) is where messages go when a consumer or connector has given up trying to process them — a malf…

Knowledge check

Q1.What does UnderReplicatedPartitions > 0 indicate?

Q2.What is the recommended approach to Kafka security in production?

Q3.During a rolling restart, what should you verify before restarting the next broker?

Q4.A DLQ topic suddenly has 25,000 new messages. What's the best first move?