What's your strategy for monitoring Kafka in production with 100+ topics?

Monitoring: 1) Collect the brokers' JMX metrics (Kafka uses Yammer Metrics; remote JMX is disabled by default, so enable it with JMX_PORT and secure it) and ship them to a metrics system; a common stack is a JMX exporter into Prometheus with Grafana dashboards, plus a lag exporter for consumer lag. 2) Alert on the metrics the Apache docs list with a normal value: UnderReplicatedPartitions (normal 0), UnderMinIsrPartitionCount (normal 0), OfflineLogDirectoryCount (normal 0), and watch OfflinePartitionsCount on the controller; RequestHandlerAvgIdlePercent and the network processor idle percent are ideally above 0.3; ISR shrink and expand rates are expected to be 0 apart from broker restarts. 3) Consumer lag: use the records-lag-max client metric or kafka-consumer-groups.sh --describe (LAG is LOG-END-OFFSET minus CURRENT-OFFSET) and alert when lag keeps growing, with thresholds taken from each consumer's SLA. 4) Per-topic views: BytesInPerSec, BytesOutPerSec and MessagesInPerSec are available per topic, so with 100+ topics use dashboard templates and alert on aggregates, then drill down per topic. 5) Watch disk usage and request total time (queue, local, remote, response send); Apache gives no disk threshold, so set one from your growth rate. 6) Review capacity regularly.

Expected key concepts: JMX, Prometheus, Grafana, consumer lag, UnderReplicatedPartitions, UnderMinIsrPartitionCount, OfflinePartitionsCount, RequestHandlerAvgIdlePercent, BytesInPerSec, kafka-consumer-groups.sh, records-lag-max