Faster, Safer, Serverless - Empowering Apache Spark Standalone Cluster on Kubernetes - Huichao Zhao

Don't miss out! Join us at our next Flagship Conference: KubeCon + CloudNativeCon North America in Salt Lake City from November 12 - 15, 2024. Connect with our current graduated, incubating, and sandbox projects as the community gathers to further the education and advancement of cloud native computing. Learn more at kubecon.io
Faster, Safer, Serverless - Empowering Apache Spark Standalone Cluster on Kubernetes - Huichao Zhao, Apple
In the realm of running quick data analysis via Spark SQL on Kubernetes, the impact of prolonged startup times is undeniable, affecting overall processing efficiency. For scenarios involving short processing tasks, any delay can cascade into hurdles, potentially disrupting entire Airflow task DAGs. In this talk, we will explore how to deliver a truly K8s-native Serverless Spark Service on Kubernetes, emphasizing speed, simplicity, with a new K8s operator for standalone cluster creation and job submission. Instead of relying solely on Spark, it also harness the elastic and police management power of Kubernetes with K8S metrics server, HPA and Kyverno, simplifying the workflow for Apache Spark itself, infra engineers, and users. The solution provides rapid responsiveness (less than 4 seconds) and facilitates the integration of longevity ML training frameworks. Join us as we propel Apache Spark into a realm of unparalleled efficiency and responsiveness, with Kubernetes as its core.