With the rapid development of Xiaomi’s business needs, particularly the ongoing implementation in scenarios like Generative AI and autonomous driving, the underlying technology stack is continuously evolving towards big data and cloud-native directions. This evolution places multiple demands on the storage system, including large capacity, high performance, low cost, and multi-protocol access. Simultaneously, business data often needs to flow across multiple systems such as object storage, high-performance file storage, HDFS, and local disks, leading to issues like complex pipelines, elevated migration and operational costs, and uneven resource utilization. To address these challenges, we have gradually implemented a unified file storage solution in our production environment to support data access, management, and distribution across various scenarios. This practice currently covers 40 clusters, supporting the management of hundreds of billions of files and a storage scale of hundreds of petabytes. This presentation will focus on this solution, detailing the following key aspects: Unified Architecture Design:​ The overall division of labor among the client, control plane, and data plane, along with a tiered design comprising the capacity layer, cache layer, and performance layer; client-side support for various access methods including POSIX, HDFS, S3, CSI Driver, and Python SDK. Storage Practice for the Full AI Pipeline:​ How to meet the requirements of different stages—raw data, training data, and model file distribution—and optimize read-intensive, write-rare scenarios through caching mechanisms. Cloud Migration Practice for the Apache Big Data Ecosystem:​ How to achieve compute-storage separation for cloud migration based on Apache Hadoop/HDFS compatibility, support the conversion of Apache Hive tables to Apache Iceberg, and enable data lakehouse scenarios with technologies like Apache Paimon. Hot Data Caching and Cloud-Native Practice:​ How to improve computational efficiency through automatic caching of hot tables, achieve compute-storage separation for cloud-migrated services via CSI mounting, and reduce disk redundancy and accelerate model loading through shared data and cache distribution. Through these practices, we aim to share insights on implementing unified storage in Data + AI environments and how to balance migration complexity, system scalability, access performance, and overall cost.

Speakers:

朝晖 王: Xiaomi Software R&D Engineer

HDFS/Ozone/JuiceFS contributor