Databricks, Delta Lake and You

Databricks, Lakes & Parquet are a match made in heaven, but explode with extra power when using Delta Lake. This session will dive into the details of how Databricks Delta works and how to make the most of it.
Speaker: Simon Whiteley SQLbits.com/speakers/Simon_Wh...
SQLbits.com/Sessions/Databrick...
Tags: Optimising,Developing,Managing,Cloud,Databricks,Python,Spark,Data Lake,Big data analytics,Modern Analytics,delta lake

[@ericbegg8727]

Simon - you couldn't be any better at explaining all these concepts. Thanks again

[@nithints302]

Accidentally hopped on to your channel I can listen for the whole day

[@Markttt5]

Hey Simon, if I was still based in the UK, I’d be knocking on your door and handing you a beer. Fantastic video (again) - this is going to help me help my organisation so much. You are by far, the best speaker, most passionate dude about data that I watch on KZbin. Many thanks.

[@mdzakariabarbhuiya1608]

This is one of the best explanation on Delta Lake!!

[@andycarter9845]

Deserves many more thousands of views. Fantastically clear.

[@chandraxg1]

Simon... thank you so much for an excellent video...

[@sunnysoni88]

I have never seen such a clear video for Delta Lake, This is just great stuff. My understanding of Delta Lake is so well now, Thanks for sharing your knowledge

[@adityajakka9856]

Great job explaining the Delta Lake, Simon. I thought you did a fantastic job with your slides and working examples. That's exactly how I look to learn new data concepts. More power to you, mate :)

[@RodrigoBocanegraCruz]

Great video. Thanks Simon!

[@nimesharya909]

awesome video, precise , clear and with easy to understand examples

[@tj_lee]

Great content, help clarifies a lot on delta tables!

[@Boompiee]

Great video as usual Simon, thank you very much!

[@manideepatalukdar9201]

Thanks you so much! This is such a clear explanation of Delta concepts!

[@jeevanb8623]

Beautifully Explained...

[@esoterictime]

Quality stuff. Subscribed!

[@denermoreira15]

just amazing

[@siddhu1076]

Wonderful explanation 🙂👍

[@simonheath8701]

I'm new to DataBricks and found this as my first video when searching. What a Gem. Haven't bothered to watch any others as it was such a great journey. As someone who spent over 30 years using SQL and saw all the big data stuff from afar I was thinking they are basically using unindexed flat files with a 16 node server cluster... hmmn, that's not advancement. Seeing how they added SQL, journalling and transaction management - I wonder how long it will take them to add indexes and create a block structured database ;)

[@SQLBits]

Lovely to hear, thank you simon! I am sure the team at kzbin.info/door/mRI-X6XoeH2dQE4BShRU9Q will love to hear this!

[@mohammedsafiahmed1639]

hey simon, when you say block structured database, you mean in opposition to traditional rdbms like sql server which are page structured db, right?

[@murtazajabalpurwala8124]

Very nice video. One of the best videos for understanding the data lake related complex issues. One recommendation is sound audibility should be improved. Thanks again for the amazing video

[@SQLBits]

Thank you for sharing your opinion! All these sessions are recorded LIVE at SQLBits in front of a crowd, so we do apologize if the audio isn't of the best quality!

[@Knigh7z]

The warehouse is also generally optimised for concurrent queries over many consumers which lake tools like Spark are not and is where Databricks SQL is closing the gap.

[@kcbonzer]

Hello Simon, this is one of the most lucid videos I have come across. Thank you conveying the message in a very simple manner. I am curious to know what you take is on Snowflake vs Databricks ! Ingestion, Storage, Architecture, Performance & Cost based comparison. A professional, unbiased & candid opinion, if you will :)

[@SQLBits]

Hey! He has a video out on our channel about 'Databricks VS Synapse Analytics' if that's something your interested in (We'll let him know about Snowflake!) kzbin.info/www/bejne/fJvWn4mrmr2coLM

[@bobhaffner5902]

Hi Simon, great video! Hey, do you know if updating a delta table via Synapse Spark NB is supported?

[@SQLBits]

Hey Bob, Simon has his own YouTUbe Channel here if it has any content you are interested in! - kzbin.info/door/mRI-X6XoeH2dQE4BShRU9Q

[@RodrigoBocanegraCruz]

Hi, is it delta suitable for tracking data changes overtime, like for examples every day? Or is it more suitable for tracking transformation changes in a given dataset? I have read is more about the second but want to check. Thanks!

[@mohammedsafiahmed1639]

deltra records every single transaction that happens to a table in the delta log. Every time a transaction happens like an update, insert delete or merge, it gets recorded as a json in the delta log. And it gets a version number. Updates and deletes do not physically update and delete the files, but just update the transaction log. This gives you the ability to travel back per transaction basis. Its pretty cool.