Processing petabyte-scale shuffle data everyday poses significant challenges for batch jobs, especially shuffle performance. In this talk, Erik will present how Apache Celeborn is used with Spark at Ant Group, dive into several topics including correctness validation, bottleneck diagnose, performance optimization and DFS integration.

Speakers:

Erik Fang, Software Engineer at Ant Group, Tech Leader