The past decade has seen a transformative evolution in Flink’s deployment paradigms, workload patterns, and hardware advancements. From the tightly coupled compute-storage nodes of the map-reduce era, we have transitioned to a cloud-native world where containerized deployments on Kubernetes are now the norm. To fully embrace this shift, Flink 2.0 introduces Disaggregated State Management, leveraging Distributed File Systems (DFS) as the primary storage medium. This architectural innovation addresses critical challenges posed by the cloud-native environment while enabling new levels of scalability, performance, and flexibility. In this session, we will introduce the the design of storage and Disaggregated State Management in Flink 2.0 and the thinking behind this, including the integration of remote state, the asynchronous state access framework, and the new state backend designed for DFS storage medium.

Speakers:

Zhaoqian (Zakelly) Lan has been contributing to open source projects for many years and is an active PMC member & committer in Apache Flink project. He specializes in streaming and key-value storage. Currently, Zhaoqian works at Alibaba Cloud, where he focuses on advancing the state management and checkpointing capabilities of Flink, driving innovation and efficiency improvements.