Kafka Streams in 4.x: Server-Side Rebalancing and Dead Letter Queues

Overview

Kafka Streams gets the idea behind KIP-848. The Streams rebalance protocol (KIP-1071) is built on the new consumer group protocol and moves task assignment to the broker. It was early access in 4.1, explicitly not for production, and reached general availability with a limited feature set in 4.2. Applications opt in with group.protocol=streams. The 4.2 and 4.3 developer guide lists what the new protocol does not support yet: static membership, warmup tasks and rack-aware assignment (only the sticky assignor is available), pattern-based topic subscription, and online migration between the classic and streams protocols. A significantly changed topology needs a new streams group. The KIP's tracking page plans static membership for 4.4 and targets warmup tasks at 4.5, with no release yet for topology updates, online migration and regex subscription. The offset reset tool for streams groups is kafka-streams-groups.sh, because kafka-streams-application-reset.sh is for the classic protocol. The developer guide also recommends against migrating from classic to streams groups on 4.2.0, and says migrations are safe from 4.2.1. Separately, 4.2 adds dead letter queue support to Streams exception handlers (KIP-1034): handlers can return DLQ records through a new Response class, with the raw source record bytes available in the error context. That builds on the processing exception handler from 3.9 (KIP-1033), which 4.3 extends to global tables (KIP-1270). 4.0 removed every Streams API that had been deprecated in 3.6 or earlier, such as KStream#transformValues, and 4.3 deprecates the streams-scala module for removal in 5.0.

Analogy

Key configuration

Opting in (4.2)properties

# Streams application: opt in to the server-side rebalance protocol
group.protocol=streams        # default: classic

# Broker: the protocol has to be in the coordinator's list.
# 4.2 default: classic,consumer,streams  (the setting is deprecated in 4.3)
group.coordinator.rebalance.protocols

# 4.0 migration: replace the removed transformValues()
KStream#transformValues  ->  KStream#processValues

# 4.3 addition (KIP-1270)
processing.exception.handler.global.enabled

Production tip

Check yourself

How does a Kafka Streams application opt in to the server-side rebalance protocol (KIP-1071)?

Sources

Checked against Kafka 4.3 documentation on Oct 2026. Draft: not yet reviewed by a Kafka specialist. Releases move on, so confirm against the current documentation before you act on a number.