How Post-4.0 Kafka Streams Reshapes Development and Operations

As Apache Kafka enters the 4.0 era, not only has the underlying architecture undergone historic transformations, but Kafka Streams has also experienced a series of evolutions in developer experience and operational mechanisms. This session will guide the audience through key updates beyond Kafka Streams 4.0, analyzing how these features fundamentally reshape the lifecycle of stateful streaming applications.

The presentation will center around three core dimensions:

1. Coordination & Operations The classic rebalance protocol has been with Kafka for a decade, and so has its central cost: the join barrier. Cooperative rebalancing (KIP-429) already removed the stop-the-world pause, but the barrier itself stayed. Every round still waits for the slowest member to check in before anyone learns their new assignment, and handing a task to another instance takes two of those rounds. KIP-848 goes after the barrier itself, by folding coordination into the heartbeat that members are already sending. The assignment is computed on the broker. Different members can sit at different points of the same reassignment. None of them is waiting on the whole group. Kafka Streams could not simply adopt it. The unit of assignment in a streams group is a task, not a partition. A task carries a state store, wants standby copies on other instances, and belongs to a topology. KIP-1071 builds on the same broker-driven design and extends it the concept: using the heartbeat mechanism to handle the rebalance.

2. Resilience & Error Handling Produce failures have always been part of running in production. The way Kafka Streams handled them has changed shape three times. In the early days your handler had two choices: drop the record or fail. KIP-572 then made retriable errors retry automatically by replaying the task, but it bypassed your handler. KIP-1065 hands that decision back to you: retriable errors now go through your handler, and you choose whether to retry, continue, or fail. When you decide to stop retrying, KIP-1034 gives the record somewhere to go — a dead-letter queue.

3. Observability If you wanted to know which step of a pipeline was slow, your only option was to wrap every lambda by hand. Some functions don’t have lambda input for measurement logic. KIP-1112 closes the gap with a single injection point: one class, one config, and every processor in the topology comes back wrapped. We will use it to get per-operator timings from a running application, and point out the one configuration mistake that makes the whole mechanism silently do nothing.

Target Audience: Whether you are a data engineer striving to streamline topology code, an architect focused on system tracing, or an SRE pursuing zero-downtime scaling, attendees will gain practical architectural insights and code examples from this session to fully master the next-generation capabilities of Kafka Streams.

Key Takeaways:

• Understand why the classic rebalance protocol behaved the way it did, what KIP-848 changed, and why Kafka Streams needed a protocol of its own on top.
• Learn how error handling in Kafka Streams arrived at its current shape, and how to build bounded retry and dead-letter behaviour with the 4.x API.
• See how to attach timing, logging or tracing to every processor in a topology without editing the topology itself.

Speakers:

PoAn Yang: ASF, Apache Kafka / YuniKorn committer

I am an open-source software enthusiast, primarily focusing on Apache Kafka. I am among the top 20 contributors, and my work involves enhancing the AsyncKafkaConsumer and developing the next-generation group coordinator.