Delta Live Tables: Building Reliable ETL Pipelines with Azure Databricks

[@SAURABHKUMAR-uk5gg]

@30:03, if you're defining the schema while creating the table, then why again selecting map(inferschema = True) ?

[@artus198]

I sometimes feel, the good old ETL tools like SSIS , Informatica were easier to deal with ! 😄 (I am a seasoned on premise SQL developer, transitioning into the Azure world slowly).

[@SAURABHKUMAR-uk5gg]

that was only good if you are working on a legacy data architecture or so called monolithic architecture. With the amount of data generation growing each day, we need SaaS like Databricks and Snowflake to perform all the data activities.

[@artus198]

@@SAURABHKUMAR-uk5gg - the whole idea of service principal Id, key , storing them in a vault , was total rubbish architecture.. now they are slowly moving towards Managed Identity... Azure is totally not worth it !

[@priyankpant2262]

Great video ! Can you share the github location of the files used ?

[@saikeerthanakattige617]

How to intially get started with databricks like creating clusters, data, notebook, how to set up the infrastructure. I am not able to move forward because of that! Please help

[@Vishnu-Kanth-01]

I guess it helps kzbin.info/www/bejne/e6qtmKyhfsislas

[@pankajjagdale2005]

crystal clear explanation thank you so much can you provide that notebook ?

[@supriyasharma9517]

can you please provide code for this?

[@anantababa]

Awesome training, can you please share the data file, i want to try it.

[@Databricks]

Nice video🤩

[@SQLBits]

🥳

[@menezesnatalia]

Nice tutorial. Thanks for sharing. 👍

[@samanthamccarthy9765]

Awesome thanks so much . this is really useful for me as a Data Architect . much is expected from us with all the varying technology

[@MohitSharma-vt8li]

Can you please provide us the notebook DBC file or ipynb.. By the way great session, Thanks

[@SQLBits]

Hi Mohit, you can find all resources shared by the speaker here: events.sqlbits.com/2023/agenda You just need to find the session you're looking for and if they have supplied us with their notes etc, you will see it there once you click on it!

[@MohitSharma-vt8li]

@@SQLBits thanks so much

[@trgalan6685]

Great presentation. No example code. What's zero times zero?

[@amadoumaliki]

As usual! Mahit wonderful!

[@germanareta7267]

Great video, thanks.

[@tratkotratkov126]

Hm … Where in these pipelines you have specified that nature of the created/maintained entity - bronze, silver or gold other then the name of the object itself. Also where these LIVE tables are exactly stored - from your demonstration it appear they all live in the same schema / database while in real live the bronze, silver and gold entities have designated catalogs and schemas.

[@prashanthmally5765]

Thanks SQLBits. Question: Can we create a "View" on Gold Layer instead having "Live Table" ?

[@supriyasharma9517]

Great video and easy explanation. I hope you come up with a series on step by step on Databricks for beginners like me who are finding to difficult / struggling to make the switch. Thanks for your efforts

[@ADFTrainer]

pls provide code links

[@Chandurkar-i2y]

For complex rule based transformations how we can leverage it?

[@Chandurkar-i2y]

Is there any way to load new files sequentially if bunch of files arrived at a time?

[@ashwenkumar]

Does delta live tables in all the layers has filesystem linked to it as like in hive or Databricks ?

[@ananyanayak7509]

Well explained with so much clarity. Thanks 😊

[@SQLBits]

Our pleasure 😊

[@ADFTrainer]

@@SQLBits Can you provide code. Thanks in advance..

[@Ptelearn4free]

Databricks have pathetic UI...

[@olegkazanskyi9752]

Is there a video on how data is pulled from the original source, like a remote SQL/noSQL server, or some API? I wonder how data is getting to the data lake? I assume this first extraction should be a bronze layer.

[@Rangapetluri]

Wonderful session. Sensible questions asked. Cool

[@lostfrequency89]

Can we create dependency between two notebooks?

[@MichaelEFerry]

Great presentation.

[@SQLBits]

Thanks for watching :)

[@guddu11000]

shoud have showed us how to trobleshoot or debug

[@thinkbeyond18]

I have a general doubt in autoloader . does autoloader required to run in a job or notebook triggering manually .Or no need to touch anything once we written the code as when as the file arrives it will run automatically and processed the files.

[@Databricks]

Trigger your notebook that contains your DLT + Auto Loader code with Databricks Workflows. You can trigger it using a schedule, a file arrival, or choose to run the job continuously. It doesn't matter how you trigger the job. Auto Loader will only process each file once.

[@walter_ullon]

Great stuff, thank you!

[@srinubathina7191]

Wow super stuff thank you sir

[@SQLBits]

Glad you liked it!

[@starmscloud]

Learned a Lot from this . Thank You for this video !

[@SQLBits]

Glad it was helpful!

[@benim1917]

Excellent

[@TheDataArchitect]

I don't get the usage of VIEWS between Bronze and Silver tables.

[@TheDataArchitect]

Anyone?

[@SQLBits]

Hi Shzyincu, you can get in touch with the speakers who taught this video via LinkedIn and Twitter if you have any questions!

[@richardslaughter4245]

My understanding (as an "also figuring out data bricks" newb: * View: Because the difference between bronze and silver in this instance is very small (no granularity changes, no joins, no heavy calculations, just one validation constraint), it doesn't really make sense to make another copy of the table when the view would be just as performant in this case. * "Live" view: I think maybe this is required because the pipeline needs it to be a live view to properly calculate pipeline dependencies Hopefully that understanding is correct, or others will correct me :) My follow up question would be: As I think about that validation constraint, it really seems like in this case it seems functionally identical to just applying a filter on the view. Is that correct? If so, is the reason to use the validation constraint rather than a filter, mostly to keep code consistency between live tables and live views?

[@anilkumarm2943]

You don't materialize as new tables evertime, We sometimes materialize it as views. So minor transformations like changing the type of the field etc.

[@freetrainingvideos]

Very well explained, Thanks