Spark Data Engineering Patterns - Shortcuts and external tables

Fabric Espresso Presents: "Spark Data Engineering Patterns - Shortcuts and external tables"
📌 About This Episode:
In today's data-driven world, mastering Spark and Big Data technologies is no longer optional-it's essential. That's why we've invited Daniel Coelho to share his invaluable insights. Daniel is driving Delta Lake at Microsoft Fabric and Azure Synapse Analytics. He's laser-focused on Data Engineering experiences using Spark and Big Data technologies. His work enables BI Analysts, DBAs, Data Engineers, and Data Scientists to shape their data effectively and build incredible solutions.
In this episode, we'll dive deep into:
📊 The Difference Between Managed Tables, External Tables, and Views
📊 Why It's Worth Using Delta Format
🛠️ Shortcuts for Optimizing Your Spark Workflows
🗄️ How to Leverage External Tables for Better Data Management
✅ PS. Demos starts from 08:28! :)
🎙 Meet the Speakers:
👤 Guest from Product Group: Daniel Coelho, Principal Product Manager
Linkedin: / danibunny
👤 Host: Estera Kot: Senior Product Manager at Microsoft
LinkedIn: / esterakot
Twitter: / estera_kot

[@azizquazi]

It is always good to provide the source file locations and the ipynb files to follow the videos. I appreciate your thoughts on this. Thanks.

[@alacostabr]

Daniel you absolutely amazing! Thank you so much for clarifying and all, all learners, create capabilities to better understand concepts correctly!

[@camilamribeiro]

Good to know!!! Thanks for sharing with us Daniel!!!

[@DanielWeikert]

Share more insights about the cost attached to shortcuts. This might be very relevant for pulling data out of e.g. S3 br

[@StanleyCruvinel]

Great Dani Bunny! I’m looking for about Shortcuts tables and how create versions or branches from a data lake. Updates, Inserts only on my data branch. Closer what ‘lakefs’ do.

[@SciCoBunny]

Thats a story we are crafting into the product right now! As you may imagine, it does require a lot of integration between catalogs and other engines. While just using the Delta shallow clone feature is a possibility, we want to do much more than that! Stay tuned!

[@7effrey]

Any place where you Can see the views that has been created in the lakehouse item?