Understanding Databricks & Apache Spark Performance Tuning: Lesson 02 - Spark Hardware

Following up on Databricks Performance Tuning with the best place to start: allocating Spark clusters. If you don't allocate sufficient resources, nothing else will fix the problem. How many nodes? How large should the driver and workers be? Do you need GPUs or CPUs? Should you use Photon? These and many more questions will be covered in detail.
Support me on Patreon
www.patreon.com/bePatron?u=63...
Slides
github.com/bcafferky/shared/b...
Referenced Video by Daniel Tomes, Databricks
• Apache Spark Core-Deep...
See the Full Playlist Here:
• Databricks and Apache ...

[@MoulayC.-qv3rn]

Thanks, very useful and up to date.

[@BryanCafferky]

You're welcome!

[@Noobsmove]

I think the main reason not to use photon, is that it costs extra^^

[@BryanCafferky]

I have not seen that there is a higher cost to using Photon but even if that's true, the time saves running the VMs should outweigh the cost. See this blog. www.databricks.com/product/photon

[@Noobsmove]

@@BryanCafferky Yeah it's a little cheeky. If you enable it, the DBU goes up. I have photon disabled for light workloads

[@BryanCafferky]

@@Noobsmove Interesting. Guess its in the fine print. 🙂