During this talk, we will present how ByteDance leverages the native engine, which is based on the open-source Gluten framework and in-house native acceleration engine Bolt, to drive significant cost saving in Spark use cases. Apache Gluten is pivotal, acting as middleware to seamlessly integrate native backends and enhance Spark performance. We will also cover the best practices for rolling it out in EB-scale data warehouses in the production environment and the optimizations on performance & compatibility, along with our future roadmaps.

Speakers:

Weiting is a senior software engineer at Intel’s Data Center and AI Group. With a decade of experience, he specializes in Big Data and Cloud Solutions. He has made significant contributions to projects like Spark, OpenStack, and recently, the Apache Gluten (Incubating) project as one of its initial committers. Among his responsibilities is harnessing the potential of hardware to enhance the performance of big data workloads.