Organizations today are overwhelmed with vast data requiring effective batch processing. Managing these jobs can lead to complexity, resource wastage, and increased operational costs, hindering business productivity.

Cloud-native technologies, including the Batch Processing Gateway, Spark on Kubernetes, and the Yunicorn Scheduler, offer robust solutions to these challenges. These tools automate job management, streamline resource allocation, and enhance scheduling efficiency. Our talk will delve into how these technologies work together to optimize processing workloads and improve operational workflows.

The result is significant: companies gain efficiency, reduce costs, and enhance their responsiveness to market needs. By adopting these cloud-native solutions, organizations can optimize operations and maintain a competitive edge in a data-driven landscape.

Speakers:

Shekhar is deeply passionate about open source software and actively contributes to various projects, including SymPy, Ruby gems like daru and daru-view (which he authored), Bundler, NumPy, and SciPy.

He successfully completed Google Summer of Code in 2016 and 2017 and has served as an admin for SciRuby, mentoring multiple organizations.

Shekhar has spoken at prominent conferences such as RubyConf 2018, PyCon 2017, ApacheCon 2020, and Community Over Code 2024, as well as numerous regional meetups. Currently, he works at Apple as a Software Development Engineer.