Advancing Spark - Databricks Delta Streaming

[@prashanthxavierchinnappa9457]

I've said it a million times. The best Spark videos on KZbin.

[@NeumsFor9]

It's because Simon and team prep beforehand and strike the right pace so that it moves just right for both newbies to absorb and seasoned vets to have a speed-of-thought experience. It's art and science.

[@manu4ever249]

The tutorial is so good probably saved me many hours or even days on my work. Thank you!

[@AdvancingAnalytics]

Thanks. That's all we ever hope to do

[@rupeshagarwal5896]

I am a big fan of your work. Really big. Noone understands problems of us better than you

[@patrickchan2503]

thank you for checkpoint explanation + everything else

[@onnovanderhorst6161]

Working for Microsoft but you're my main source of information :D

[@diogodallorto1]

You are helping me a lot with this stuff! Just keep going! Thank you!

[@rohithamaz8767]

Your explanation is awesome!! i am trying it now... Just wanted to explore a lot of Streaming use cases, It would be great if you can put across some challenging streaming examples..

[@fgnb-md4ln]

Great video! Love your clear explaination. BTW Where can I find your notebook? I noticed that your github updated 4 years ago.

[@deepjyotimitra1340]

Thank you for such an amazing video. Nice explanation

[@ravikumarkumashi7065]

Very helpful vedio. I have a quick might sound silly. We have delta table partitioned by business date and it is being used as a streaming source. Now we have requirement to repartition the table by date and another column and when we do that our streaming query does not detect inserts or updates. Do i need to clean the checkpoint ? Setting starting version does not work for us.

[@HiteshTulsaniAtGooglePlus]

You got a new sub from Aus. Epic stuff mate! Helps a lot.

[@sid0000009]

Thanks ! do you also share a Github where we can pick your notebooks?

[@sumashruthika7852]

Great video!

[@rahulsood81]

Can you explain the difference between using Auto-loader and Structured Streaming (readStream/writeStream) Also, when/how to use foreachBatch in Databricks ??

[@anilpeyyala3332]

Thanks a lot with these videos. They help a lot. I got a question though, how can we perform Stream-Stream joins and Stream-Static data joins? I have a need to multiple joins with multiple streaming sources each arriving at separate times?

[@adyashreemahapatra370]

Thank you for the video. Can we process this streaming data using readStream and writeStream inside a .py script in databricks. As, I am getting an error while reading data from the location where data is written using writeStream() function?

[@nmounika7927]

Hi, i want to trigger the data bricks whenever there is a new file loaded in delta table, do we have any option for that?

[@tanushreenagar3116]

awesome content

[@edamextreme]

The tutorial was incredible. But I have a question, can I stream from a view?

[@AdvancingAnalytics]

Hrrrrrrrm. Good question - don't think so but I've not tried it. You can use Views in Delta Live Tables continuous mode, which makes me think it's possible, but maybe only with some manual coding around to bring the elements together! I'll add it to my "things to try" list :)

[@MrDeedeeck]

Hi, not really clear exactly how different the _checkpoint file is from the _delta_log files. It seems like you could run your entire example using just _delta_log and specifying the version of the delta table to read from?

[@RubenAlvarezMtz]

Check this video out, they clearly explain the difference and how they work together kzbin.info/www/bejne/Z5SldXqpiMeqiKM

[@abhijeetnaib]

If there are lot of files at the source then exactly once semantics might not help considering the listing files will take a lot of time

[@quentindelignon1587]

Hey ! i dont know if you still answer questions but does this apply to any kind of source ? can we get this versioning and increment processing on any source like HDFS ?

[@uscgpsu1]

Great video very helpful

[@besafal1]

I am trying to write a streaming application and need to create a new df by taking every event as input. Now since spark is not available on executor so it errors or every time. Can you please help how can we create df from a separate source in streaming application

[@samk_jg]

its very helpful, thanks!

[@Sunkarakrwutarth9389]

Thanks For the video

[@HughMcBrideDonegalFlyer]

Is the notebook available to experiment with

[@AdvancingAnalytics]

Not yet! The notebooks are all fairly heavily plumbed into our training, which we can't give away. We'll figure out the best way to share, how much we want to make available etc. Just needs some free time to pull things apart :)

[@sonamjain4567]

Thanks a lot for sharing these videos. I have gone through the series of videos and its been really very helpful. I have got 1 doubt. Can you please let me know what is the difference when we write checkpointLocation as you have shown in this video vs autoloader. I have seen that in auto loader also, we give checkpoint. Are these referring to the same feature or there is any difference. Kindly help. Thanks!

[@AdvancingAnalytics]

Hey! Yep, the checkpoint mechanism is the same between autoloader and Delta streaming, although it holds slightly different metadata depending on the streaming source - delta sources hold the latest delta version that was read, autoloader sources hold the details of the queue it should reference & the files read so far.

[@sonamjain4567]

@@AdvancingAnalytics Thanks for quick reply. So what are the use cases for using each approach? Are there any pros and cons of both the approaches? Kindly suggest

[@AdvancingAnalytics]

@@sonamjain4567 I use them very differently - autoloader is good for bringing source data into the lake from external sources, delta is for change in between layers of your lake. It's less of a pro/con between the two, more depends which scenario you are trying to implement!

[@PersonOfBook]

Does it only work with delta tables or also with plain text files in the data lake?

[@AdvancingAnalytics]

You can use standard spark file streaming over files in the lake, although it can slow down over time as it scans the entire directory each micro-batch. Delta streaming is way more efficient as it can use the transaction log to isolate only new files

[@crazybauns]

i am new to databricks and streaming so apologies for the dumb question you create a table in your databricks db which gets its data from your delta file you then insert data to that table your streaming df is mapped to your delta files when you insert to your table; why is the data inserted to your delta file? the table and delta are always mapped to one another? even if you dont make it explicit, inserting to a table means inserting to a delta file?

[@AdvancingAnalytics]

No dumb questions - we're all learning! The "table" is just a logical pointer to the delta files, not like a physical table in a SQL database. Each time you query the table, it's going to the delta files regardless. So inserting into the table, or into the delta files, does the same thing. That said, you can create those logical/external tables over other files/folders if you don't want to use delta. Simon

[@crazybauns]

@@AdvancingAnalytics Thank you!

[@qweasdzxc2007]

Going further in this question: if i manually copy file in folder, where delta table already resides, will i get information from this new file also with old rows, when i select from this delta table?

[@lukasu-ski4325]

Is it really a big difference between writeStream with option trigger(once=True) and a classic run of write.format("delta").mode("overwrite")? Don't we overcomplicate it with streaming? It seems for me we can achieve the same thing with batch with less complexity. Let me know your thoughts!

[@AdvancingAnalytics]

Yeah, absolutely streaming makes it more complicated, but streaming from a delta table means you don't have to work out what has changed, it automatically picks up only new files. So you're reducing complexity of change detection, but at the price of additional complexity through streaming. Both approaches have pros/cons - for me, it's another useful tool in my toolbelt! Simon

[@muchirajunior]

great

[@user-eg1ss7im6q]

how did you write to data lake storage? .save("/mnt/xxx") did not work for me?