Spark Compute in Fabric Data Engineering and Data Science - Starter Pools vs Custom Pools Unveiled!

Fabric Espresso Presents: " Spark Compute for Microsoft Fabric Data Engineering and Data Science - Starter Pools vs Custom Pools Unveiled! " 🔥🔥🔥
📌 Navigating the world of Microsoft Fabric's Spark Compute platform? Confused about whether to go for Starter Pools or Custom Pools? Look no further! This video breaks down the distinct advantages of both, tailored to meet different needs in Data Engineering and Data Science. Whether you're looking for quick insights or need to handle resource-intensive tasks, we've got you covered!
📚 What You'll Learn:
✅ Starter Pools: Designed for quick, hassle-free access to Spark sessions. Learn how they initialize in just 5 to 10 seconds and offer always-on Spark clusters that dynamically scale based on your needs.
✅ Custom Pools: Get a deep dive into how Custom Pools offer a more tailored experience, allowing you to specify node sizes and numbers, ideal for complex and resource-intensive jobs.
✅ Billing: Understand how you're only charged for active job runs in both Starter and Custom Pools, not for setup or idle time.
✅ Choosing the Right Pool: Tips on how to choose between Starter and Custom Pools based on your specific project requirements.

[@rockinsathya]

Looks quite interesting!

[@keen8five]

Awesome, I love the single node functionality ❤

[@VikashKumar-qt3sz]

What will be use of choosing low number of machines. I know i am going to be charge same even if i use max no of machine

[@manassehkisame6286]

As of March 2024,am using an organizational SKU name:FT(Trial Capacity) with CU of 64...this was running under 5second for a long time until now when the compute time is averaging 2-3minutes.I have optimized the spark pool instance to no avail.Microsoft documentation had noted the problem last year but for lower SKU's(F2,F1).Kindly advise.