Tech Talk | Diving into Delta Lake Part 1: Unpacking the Transaction Log

[@Databricks]

Check out the Online Meetup playlist for video recordings of these tech talks. This one will be available later today! - dbricks.co/youtube-meetups

[@Databricks]

- Watch Part 2, Enforcing and Evolving the Schema: @ - Watch Part 3: How do DELETE, UPDATE and MERGE work: kzbin.info/www/bejne/bZbanpaap96fqaM

[@machinelearninginreallife3558]

From what you've said, I understand that data versioning is not a real component of DeltaLake. What we want is only to avoid some mistakes (mistaken delete). Am I right?

[@dheerajkumarsolanki5716]

There is default 30 days of transaction log retention period. So, after 30 days the older transaction logs files are automatically deleted? Similar to this, what happened to logically deleted data files after deletedFileRetentionDuration period, is they are automatically deleted or we have to manually delete it?

[@no_more_free_nicks]

No, not everybody knows what a Data Lake is, so thanks for explaining it briefly.

[@machinelearninginreallife3558]

I'm not sure to understand. Can we keep data for more than 30 days? Is it a bad practice? Is it even possible?

[@amitjaju3351]

Hello Burak, I need one small help .Could you please tell me if we are performing delete operation on Delta table which later and if we need to keep of records which we deleted then can we do that from transaction log folder or is there any other way bh which we can keep track like which record we deleted if someone ask us in future. Waiting for your response. Thanks

[@TheIceSpinner]

You don't mention how you actually store reads and writes. Do you store them differentially, and if so, what is the unit? So eg. when you delete a single row in a dataframe, is it only the deleted row that's stored in the new parquet, with some kind of flag, or the whole dataset is duplicated (minus that row)?

[@dennyglee]

There are new Parquet files that are created so that way you can have time travel. You can see which Parquet files are created within the transaction log.

[@stuckinamomentt]

So Vacuum does not remove log files (due to GDPR), then when are the log files cleaned up to avoid growing indefinitely?

[@dennylee4934]

That's correct, VACUUM does not remove the logs - only the data (parquet) files. Note that the logs are converted from JSON to Parquet which subsequently improves the performance of reading the log.

[@stuckinamomentt]

@@dennylee4934 Thanks, and I believe delta.logRetentionDuration controls how to clean up the logs

[@harikrishnasiliveri1364]

@@dennylee4934 In that case, once we do repartition, we cant achieve time-travel? (since logs are not pointing to the data files anymore) is that correct?

[@bhanu4j]

Can you share the link for this python notebook. I did not find it.

[@dennyglee]

The notebook link is hiding in the description - here you go: github.com/dennyglee/databricks/blob/master/notebooks/Users/denny.lee%40databricks.com/Delta%20Lake/Diving%20Into%20Delta%20Lake:%20Unpacking%20The%20Transaction%20Log.py

[@ArturSukhenko]

my_table/date=2019-01-01. Parquet doesn't support date format :) So date is string there?

[@kevingomez-yo3or]

date='2019-01-01'.patquet

[@Sarmoung-Biblioteca]

Otimo Video !! Obrigado !!

[@kevingomez-yo3or]

Can we have the slides?

[@dennylee4934]

Sure, you can find them in our tech-talks repo at: github.com/databricks/tech-talks/tree/master/2020-03-26%20%7C%20Diving%20into%20Delta%20Lake%20-%20Unpacking%20the%20Transaction%20Log

[@irochkalviv]

Burak Yavuz, A couple of corrections: 1. Turks arrived in Anatolia (from Central Asia), starting the 11th century, no need to falsify history with the intention of giving some rights to occupy Anatolia and the rest of so called "Turkey" 2. 1453 is also the beginning of long five centuries of abuse, looting, oppression and massacres of the native Christian populations 3. A very important important date seems to be omitted: !915, the genocide of native christian populations by the Turks Actually, the Turkic tribes that invaded Anatolia have many similarities with the fighters of the Islamic state ISIS. Is it why so called "Turkey" supported them?