Today’s AI isn’t slow — a single request returns results in seconds. But this “online real-time” is fundamentally passive inference triggered by requests, while the real world produces data as a continuous stream of events. When AI needs to perform continuous, incremental, and stateful processing over real-time data streams, we enter a different paradigm — Streaming AI. It raises distinctive challenges: treating multimodal data as a first-class citizen in streaming pipelines; supporting locally-executed computation such as video decoding, image processing, and small-model inference with GPUs inside the streaming runtime, rather than detouring through remote services; integrating agent-style multi-step reasoning, tool use, and context and memory management with event-driven stream processing; and leveraging stream processing’s heritage in joins, aggregations, state management, and temporal guarantees, rather than rebuilding it from scratch.

Flink is addressing these challenges through a series of advancements: multimodal data processing for audio, video, images, and vectors as first-class data; GPU scheduling for local computation within the Flink runtime; Flink Agents that natively embed agents with context management, memory, and tool invocation into stream processing; and a DataFrame API that offers a unified, concise interface for AI and data engineers. Combined with Flink’s core strengths in streaming state and temporal semantics, these advancements define Flink’s new role — the unified foundation for Streaming AI.

This talk uses a real-time AI sports commentary system built on Flink as an end-to-end case study: from live-stream frame extraction and visual understanding, to context-aware commentary generation, to TTS synthesis synchronized with the broadcast. A seemingly simple “AI commentator” exercises every one of these capabilities at once.

Attendees will leave with a clear view of Streaming AI’s core challenges, Flink’s latest AI capabilities, a reusable multimodal pipeline architecture, and engineering insights for balancing latency, cost, and model quality in production.

Speakers:

Xintong Song: Apache Flink PMC Member, Staff Software Engineer @ Alibaba Cloud

Xintong is an Apache Flink PMC member and a main promoter of the Flink Agents sub-project. He is also a Staff Software Engineer at Alibaba Cloud, leading a team that works on Flink’s AI and Lakehouse integrations. Prior to that, he received a Ph.D. degree in computer science from Peking University.