The rise of AI workloads—autonomous driving and large language models—is fundamentally reshaping data processing paradigms. Data has shifted from structured tables to petabyte-scale multimodal files; processing granularity has narrowed from partition-level to frame/document-level; and task dependency graphs have evolved from linear ETL pipelines into complex workflows with dynamic parallelism and conditional branching. Traditional scheduling systems, built around structured data, static DAGs, and homogeneous resources, are facing unprecedented challenges.

This talk draws on Xiaomi’s production experience of orchestrating millions of task instances daily across its autonomous driving and large language model businesses. We examine four core challenges—dynamic execution beyond static DAGs, heterogeneous resource orchestration across Spark/Flink/Script on YARN/K8s/GPU, million-scale scheduling throughput, and data-event-driven multi-condition triggering—and present our solutions. Key highlights include declarative
YAML-based workflow definitions, an architecture upgrade that delivered a 10x improvement in scheduling throughput, resource profiling and pool-based governance that boosted utilization by 50%, and our exploration of Agent-driven development and operations.

Speakers:

LiWei Yang: Software R&D Engineer

Xiaomi Senior Software R&D Engineer, leading Data Engineering at the Big Data Development Platform. Contributor to Kestra, Apache Gravitino, and Apache DolphinScheduler.