Spring Boot with Kafka
The implementation path: 20 topics across 4 tiers, from the first dependency to a production configuration checklist. The code is illustrative: types such as OrderEvent stand for your own. Class names, attributes and properties were checked against the Spring for Apache Kafka 4.1 and Spring Boot 4.1 documentation, and each topic lists its sources.
Start hereMaven/Gradle Dependencies0 of 20 topics done
- BeginnerGetting Started0/5
- IntermediateProduction Patterns0/5
- AdvancedAdvanced Patterns0/5
- EnterpriseEnterprise Architecture0/5
Getting StartedBeginner
0 of 5 doneCore setup — dependencies, basic producer/consumer, and listener configuration.
- Maven/Gradle DependenciesAdd spring-kafka to your project. Spring Boot's dependency management determines the Apache Kafka client versions, so you do not set them yourself; the kafka.version property overrides them if you need to.
- application.yml ConfigurationCore Spring Kafka properties. Spring Boot passes them to the Kafka clients and auto-configures KafkaTemplate and the listener containers from them. Kafka's own client settings go under properties.
- Basic Producer with KafkaTemplateKafkaTemplate is auto-configured: inject it. send() returns a CompletableFuture<SendResult>, and SendResult gives you the record metadata (partition and offset).
- Basic @KafkaListener Consumer@KafkaListener creates a managed listener container that polls Kafka for you. To receive an Acknowledgment and commit when you decide, use a manual acknowledgement mode (MANUAL or MANUAL_IMMEDIATE); in the other modes the Acknowledgment parameter is null.
- Topic Creation with KafkaAdminSpring creates topics at startup from NewTopic beans via KafkaAdmin; TopicBuilder keeps the declarations short. If a topic already exists the bean is ignored, except that KafkaAdmin increases the partition count when the existing topic has fewer partitions than declared. It does not change existing topic configuration unless spring.kafka.admin.modify-topic-configs is enabled.
Production PatternsIntermediate
0 of 5 doneError handling, manual acknowledgement, JSON serialization, transactions, and batch processing.
- DefaultErrorHandler with Dead Letter TopicDefaultErrorHandler is the container's default error handler. It retries a failed record by seeking back to it, following a BackOff, and when retries are exhausted it calls a recoverer. The default is FixedBackOff(0L, 9) and logging; DeadLetterPublishingRecoverer publishes the failed record to <topic>-dlt (the same partition, by default).
- @RetryableTopic (Non-Blocking Retry)A failed record is forwarded to a retry topic and the listener goes on with the main topic, instead of retrying in place. The retry-topic consumer pauses its partition until the retry is due. Ordering guarantees are lost, and the feature does not work with batch listeners or container transactions.
- Transactional Kafka ProducerA Kafka transaction writes to several topics and partitions atomically: they are all committed or all aborted. Set a transaction-id-prefix and Spring Boot configures a KafkaTransactionManager and wires it into the listener containers.
- Batch Listener PatternA batch listener receives all the records one poll() returned, in one call. With the BATCH ack mode the offsets are committed once the whole batch has been processed. RECORD ack mode is not supported for batch listeners, and neither are non-blocking retries.
- Consumer with Message HeadersKafka records can carry headers: small key-value metadata such as a trace ID or an event version. Spring can inject the record's topic, partition and offset, and your own headers, into listener parameters.
Advanced PatternsAdvanced
0 of 5 doneKafka Streams, Avro, transactions with a database, consumer lifecycle hooks, and complex error handling.
- Avro Serialization with Schema RegistryAvro with Confluent Schema Registry registers each schema and checks every new version for compatibility with the previous ones, using the subject's compatibility type (BACKWARD by default). The serializers are Confluent's KafkaAvroSerializer and KafkaAvroDeserializer; they are not part of Spring or Kafka.
- Kafka Streams with Spring BootDeclare @EnableKafkaStreams and Spring Boot builds the Streams configuration from the spring.kafka.streams properties; a StreamsBuilder is available to inject. Kafka's own Streams settings go under spring.kafka.streams.properties, where dotted keys are written in brackets.
- ConsumerSeekAware — Offset ControlA listener that implements ConsumerSeekAware gets a callback to seek the consumer: to a timestamp, an offset, the beginning or the end. Useful for replaying a time range or starting from a known position.
- Kafka and Database Transactions TogetherSpring runs a Kafka transaction in the listener container and a database transaction from @Transactional. They are synchronized, not atomic: the database commits first and then Kafka. If the Kafka commit fails, the record is redelivered, so the database update must be safe to repeat. Inside Kafka alone, a container transaction gives read-process-write exactly-once: the offsets are sent to the transaction.
- Kafka Consumer Lifecycle & Health ChecksListener containers publish events (for example when they are idle) and can be paused, resumed and inspected through the KafkaListenerEndpointRegistry, which Spring Kafka declares for you.
Enterprise ArchitectureEnterprise
0 of 5 doneOutbox pattern, Saga choreography, CQRS, multi-tenant Kafka, and production hardening.
- Outbox PatternA service cannot update its database and publish to Kafka in one transaction: Kafka cannot take part in a distributed (XA) transaction. The outbox pattern writes the event to an outbox table in the same database transaction as the business change, and a separate relay, here Debezium reading the database's transaction log, publishes it to Kafka afterwards. Delivery to Kafka is at-least-once.
- Saga Choreography with KafkaIn a choreographed saga each service reacts to events and emits its own, with no central orchestrator. This sketch shows the flow, not a production recipe: each service here writes to its database and then sends to Kafka, which is the dual-write problem the outbox pattern exists to solve.
- CQRS with Kafka and Read ModelsSeparate the write (command) path from the read (query) path. Commands produce Kafka events; consumers build read models in stores shaped for queries (a search index, a cache, a relational table). This is an architecture pattern, not a Kafka feature: Kafka carries the events.
- Multi-Tenant Kafka ConfigurationOne way to isolate tenants is by topic name (a tenant id in the topic) and, where isolation matters more, a consumer factory per tenant. A topicPattern subscription lets one listener cover all tenants' topics; the group discovers new matching topics through periodic metadata refreshes (metadata.max.age.ms, 5 minutes by default).
- Production Hardening ChecklistSettings worth writing down explicitly in a production profile. Where a value is the Kafka 4.2 default it is marked as such: restating a default documents the intent, it does not change behaviour.