As Apache Kafka enters the 4.0 era, not only has the underlying architecture undergone historical transformations, but Kafka Streams has also experienced a series of evolutions in developer experience and operational mechanisms. This session will guide the audience through key updates beyond Kafka Streams 4.0, analyzing how these features fundamentally reshape the lifecycle of stateful streaming applications.

The presentation will center around three core dimensions:

1. Topology Optimization & Data Mastery We will explore how to write more lightweight and expressive DSL topologies. With KIP-1104, developers can extract foreign keys directly from either the Key or Value, drastically reducing unnecessary data duplication and state store overhead. Additionally, we will analyze KIP-1271 and KIP-1285 (Headers-Aware State Stores). Previously, to prevent the loss of metadata like Trace IDs or tenant tags within stateful nodes, developers were often forced to “cram” this system information into pure business payloads (the Value), resulting in bloated data structures and redundant code. These two proposals finally enable state stores to natively retain Headers, allowing developers to cleanly decouple business logic from system information.

2. Resilience & E2E Observability In production environments, network fluctuations and transient failures are inevitable. We will demonstrate how to utilize the native RETRY mechanism in ProductionExceptionHandler introduced by KIP-1065 to build self-healing processing pipelines. Following this, we will combine KIP-1112 (Custom Processor Wrapping) with headers-aware state stores to showcase how OpenTelemetry tracing logic can be seamlessly injected across the entire topology, achieving true End-to-End (E2E) Tracing across stateful nodes.

3. Seamless Scaling & Operations Finally, we will address the operational pain points of scaling large clusters. We will introduce how KIP-1106 (Duration-Based Offset Reset) helps developers precisely control the data replay range when dealing with infinite-retention topics in Tiered Storage, avoiding meaningless computational waste. We will also dive into KIP-1071 (Streams Rebalance Protocol), exploring why the previous Cooperative Rebalancing still encountered “Sync Barrier” wait latencies and processing-timeout-induced “Rebalance Storms” in large clusters. By dissecting the newly introduced server-side driven assignment and independent Background Heartbeat mechanism in KIP-1071, we will show how Kafka 4.0 completely eradicates performance jitter during scaling, leading developers into an era of truly seamless background scaling.

Target Audience: Whether you are a data engineer striving to streamline topology code, an architect focused on system tracing, or an SRE pursuing zero-downtime scaling, attendees will gain practical architectural insights and code examples from this session to fully master the next-generation capabilities of Kafka Streams.

Key Takeaways:

• Understand how to use the new APIs to eliminate redundant conversion logic through practical DSL code examples, allowing streaming applications to return to a clean, maintainable business essence.
• Master how to elegantly integrate End-to-End (E2E) Tracing and native error retry mechanisms without polluting business code, elevating system observability in production environments.
• Gain a thorough understanding of the underlying mechanisms of server-side rebalancing and independent heartbeat threads, empowering your team to make more confident architectural decisions during future Kafka upgrades, routine deployments, or large-scale cluster expansions.

Speakers:

PoAn Yang: ASF, Apache Kafka / YuniKorn committer

I am an open-source software enthusiast, primarily focusing on Apache Kafka. I am among the top 20 contributors, and my work involves enhancing the AsyncKafkaConsumer and developing the next-generation group coordinator.