Spring Boot with Kafka
The implementation path: 20 topics across 4 tiers, each with working configuration and code you can copy into a real project — from the first dependency to the production checklist.
Getting StartedBeginner
Core setup — dependencies, basic producer/consumer, and listener configuration.
- Maven/Gradle DependenciesAdd spring-kafka to your project. It transitively brings in kafka-clients. No explicit kafka-clients version needed — spring-kafka manages it.
- application.yml ConfigurationCore Spring Kafka properties. These map directly to Kafka client configs. Spring Boot autoconfigures KafkaTemplate and listener containers from these.
- Basic Producer with KafkaTemplateKafkaTemplate is autoconfigured. Inject it directly. send() returns a CompletableFuture in Spring Kafka 3.x (ListenableFuture in 2.x).
- Basic @KafkaListener Consumer@KafkaListener is the Spring Kafka annotation for creating managed consumer containers. The container handles threading, error handling, and offset commits.
- Topic Creation with KafkaAdminSpring creates topics at startup via KafkaAdmin. Use TopicBuilder for concise declarations. Topics are created if missing, not modified if existing.
Production PatternsIntermediate
Error handling, manual acknowledgement, JSON serialization, transactions, and batch processing.
- DefaultErrorHandler with Dead Letter TopicDefaultErrorHandler retries failed messages with configurable backoff, then routes to DLT after exhausting retries. Essential for production consumers.
- @RetryableTopic (Non-Blocking Retry)Non-blocking retry is superior to blocking retry. Failed messages go to retry topics; the main consumer continues processing without pausing. Strongly preferred for production.
- Transactional Kafka ProducerKafka transactions enable atomic writes across multiple topics/partitions. Required for exactly-once when combining multiple Kafka writes with a single business operation.
- Batch Listener PatternBatch listeners receive a full poll() batch at once. More efficient for bulk processing — fewer method invocations, better throughput for high-volume topics.
- Consumer with Message HeadersKafka message headers carry metadata: trace IDs, event type, schema version, retry count. Essential for distributed tracing and routing.
Advanced PatternsAdvanced
Kafka Streams, Avro, exactly-once, consumer lifecycle hooks, and complex error handling.
- Avro Serialization with Schema RegistryAvro + Schema Registry is the production standard for strong typing, schema evolution, and compact wire format. Spring Kafka integrates seamlessly via ConfluentKafkaAvroSerializer.
- Kafka Streams with Spring BootSpring Boot autoconfigures Kafka Streams when spring.kafka.streams properties are set. Use @EnableKafkaStreams and inject StreamsBuilder.
- ConsumerSeekAware — Offset ControlConsumerSeekAware gives fine-grained control over consumer position. Useful for replaying specific time ranges, starting from a known position, or implementing custom seek logic.
- Exactly-Once with ChainedTransactionManagerCombine Kafka transactions with database transactions for exactly-once consume-transform-produce semantics. Requires careful configuration.
- Kafka Consumer Lifecycle & Health ChecksMonitor listener container state, implement lifecycle hooks, and integrate with Spring Boot Actuator health indicators.
Enterprise ArchitectureEnterprise
Outbox pattern, Saga choreography, CQRS, multi-tenant Kafka, and production hardening.
- Outbox Pattern (Zero Dual-Write Risk)The Outbox Pattern solves the dual-write problem. Write to DB and outbox table in one ACID transaction. A Debezium CDC connector reads the outbox and publishes to Kafka. Guarantees no orphaned DB writes without events.
- Saga Choreography with KafkaChoreography-based Saga uses domain events to coordinate multi-service transactions without a central orchestrator. Each service reacts to events and emits its own.
- CQRS with Kafka and Read ModelsSeparate write (command) and read (query) paths. Commands produce Kafka events. Consumers materialize events into optimized read stores (Elasticsearch, Redis, RDBMS).
- Multi-Tenant Kafka ConfigurationIsolate tenants via topic namespacing, dynamic consumer factory, and per-tenant listener containers. Supports both shared and dedicated cluster architectures.
- Production Hardening ChecklistEssential production configurations that prevent common failures in Kafka-based Spring Boot services.