Tech Talk | Diving into Delta Lake Part 3: How do DELETE, UPDATE, and MERGE work

[@weldervicentemoreiramartin9467]

Hello, I couldn't replicate the delta table upsert even following the documentation. Does not update or insert new records. I opened a request on the databricks forum. I opened a community discussion forum called: Delta lake upsert - databricks community I couldn't post the forum link here on youtube, the post is deleted if there is a link.

[@johndoes461]

Great detailed info TD and Denny.

[@NeerajGarg]

Thank you for sharing detailed information on the internals of data lake

[@thomsondcruz5456]

Enjoyed the session. Delta is awesome. Also, Denny looks like Nate Shelly from Ted Lasso.

[@rakeshdey1702]

Great Session.. So We can not do any saprk.sql() operation for delta lake from EMR?? only have option using databricks for spark 2.4?

[@dennyglee]

You can update your EMR instance to utilize Spark 2.4.

[@vaasumusic7]

can we use certain partitions in clause condition so that merge/insert/update happens only in that partition ?

[@NameEncrypted]

In case of SCD 2, If there is a delay in data so we skipped one day data load and loaded few more days. Is it possible to travel back in-time towards left and do the merge action? And also by keeping remaining data towards right. Can you give some examples bon this?

[@somily800]

What's better way load all the data from the data lake into dataframe and create a delta table and read this delta table with dataframe or it's the same using SQL delta table, for example I read data from data lake with 2 billion of rows at least what is the way to add only the new data from data lake to my delta table and sometimes for rules of business it's necessary to replace al my data what it's the better way ?

[@Kirbys911Heaven]

This is super helpful. Thank you.

[@mandrakeguy88s95]

Hi please let me know how to query like database table using some tools and not using program whatever operation you did in programmatically

[@berkerkozan3659]

If I delete a record from my bronze file, can I also delete it also from silver table through structured streaming? Or should I explicitly delete it from silver and gold tables one by one?

[@tarakasep]

How to work on composite key columns

[@nasirmehmoodpanwar877]

Cool Stuff

[@sreeramgarlapati9024]

nice talk TD and Denny. love this. regarding the problem statement of improving performance of writes in SparkOutputMode.Update - I belv. has 2 parts: 1) accelerate the algo. to locate the record to be updated 2) reduce the write overhead to update in the end: update=delete+insert at a row/record level. or at the file level. Right now - this is implemented at file level. can we bring this to record level.?

[@funwithazure1861]

Great job Guys! Are the notebooks and slides available for download? On Git some where? If yes, please paste a link...Cheers...

[@shaifaslam1600]

Tathagata Das's cursor is freaking me out, I don't know how many times I have wiped my screen because of that.. XD