How to Build a Delta Live Table Pipeline in Python

Delta Live Tables are a new and exciting way to develop ETL pipelines. In this video, I'll show you how to build a Delta Live Table Pipeline and explain the gotchas you need to know about.
Patreon Community and Watch this Video without Ads!
www.patreon.com/bePatron?u=63...
Useful Links:
What is Delta Live Tables?
learn.microsoft.com/en-us/azu...
Tutorial on Developing a DLT Pipeline with Python
learn.microsoft.com/en-us/azu...
Python DLT Notebook
learn.microsoft.com/en-us/azu...
DLT Costs
www.databricks.com/product/pr...
Python Delta Live Table Language Reference
learn.microsoft.com/en-us/azu...
See my Pre Data Lakehouse training series at:
• Master Databricks and ...

[@VeroneLazio]

Great job as always Bryan, keep it up, you are helping us all!

[@BryanCafferky]

Thanks Verone.

[@dhruvsingh9]

Wonderful demo. Thanks

[@BryanCafferky]

You're welcome.

[@balanm8570]

Really great content to understand in detail about how DLT works. Thanks @Bryan for your effort in making this video.

[@BryanCafferky]

You're welcome!

[@stu8924]

Another awesome tutorial, thank you Bryan.

[@BryanCafferky]

You're Welcome!

[@user-pz5eh7uh7n]

2:40 It seems like Premium is required for most features now, as everything is based on Unity Catalog which in turn is a premium feature.

[@realjackofall]

Thanks. This was useful.

[@karolbbb5298]

Great stuff!

[@jkarunkumar999]

Great explanation,Thank you

[@BryanCafferky]

You're Welcome!

[@satyajitrout8670]

Great one Bryan. Super Video

[@BryanCafferky]

Thanks

[@Thegameplay2]

Really useful

[@user-es5ih7wy1u]

Hello Bryan Sir, Thanks for your amazing videos.

[@BryanCafferky]

HI Ibrahim, Thanks. Did you watch the video? I explain about that.

[@gatorpika]

Great video. Like how you dive into other topics like should we use it? What does it cost? It's running extra nodes in the background....etc. Lot of useful info in your explanations. Just wanted to mention on the expectations not having a splitter to an error table, we had a demo from Databricks recently and their approach was to create a copy of the function with the expectation, but pointed at the error table and with the inverse expectation of the main function. I mentioned this wasn't ideal since you would have to run the full job twice and they didn't have much to say. We have a different approach to dealing with errors so not a huge deal from our standpoint, but still not great in general.

[@BryanCafferky]

Thanks for the feedback and your experience with expectations.

[@jeanchindeko5477]

Thanks for this video Bryan. 13:27 if you want to quarantine some data based on a given rule, the workaround is to create another table and put an expectation to drop all the good records and keep only the bad one

[@amarnadhgunakala2901]

I love your video consistent

[@BryanCafferky]

Thank You!

[@krishnakoirala2088]

Thanks for the awesome video! A question if you could help: How to do CI/CD with delta live tables?

[@BryanCafferky]

This blog explains it www.databricks.com/blog/applying-software-development-devops-best-practices-delta-live-table-pipelines

[@krishnakoirala2088]

@@BryanCafferky Thank you!

[@JustBigdata]

Hi. Just wanted to make sure something. I am using Azure databricks where I already have two clusters in production. Now, if I want to create a DLT pipeline (assuming that's the only way to use Delta live tables ), would that create a new cluster/compute resource ?

[@mateen161]

Would it be possible to create unmanaged tables with a location in datalake using DLT pipelines ?

[@wrecker-XXL]

Hey Bryan, Thanks For the video. Just curious, do we know the list of decorators which we can use in DLT pipelines. I looked into the documentation but was unable to find it

[@BryanCafferky]

Since you have the dlt package, you have the code so you should be able to inspect the modules using Python functions like dir() or even view the code, see stackoverflow.com/questions/48983597/how-to-print-source-code-of-a-builtin-module-in-python DLT doc is here docs.databricks.com/en/delta-live-tables/python-ref.html#:~:text=In%20Python%2C%20Delta%20Live%20Tables,materialized%20views%20and%20streaming%20tables. I've not tried these things on dlt so let me know how it goes please.

[@ShubhamSingh-ov1ye]

what I have observed, the materialized view is recomputing everything from scratch, what can we do to do incremental ingestion into the materialized view based on the group by clause if we provide.

[@MOHITJ83]

Nice info! Is is a bad design to have bronze, silver and gold layer in the same schema. I believe DLT doesn’t work with multiple schemas

[@ezequielchurches5916]

hey bryan, great video, I have a quick quesiton, when you create a DLT for RAW, PREPARED and the last layer, that tables are created in the lakehous into BRONZE< SILVER AND GOLD?

[@BryanCafferky]

Yes, if I understand you. You can direct the tables to fit into the medallion architecture. See www.databricks.com/glossary/medallion-architecture

[@user-sp5yi7lc9p]

Hi Bryan, Is it possible to use Standard cluster to create Delta live tables instead of creating new cluster every time ?

[@BryanCafferky]

I don't see coverage of that in the docs but here's the link to check yourself. learn.microsoft.com/en-us/azure/databricks/delta-live-tables/settings You may be able to create a workflow with your own cluster and call a DLT pipeline. Not sure if that will still create a separate cluster.

[@TheDataArchitect]

Really confused if i use DLT's for my project or old way of doing it for Medallion architecture. Now i watching your video, that DLT's cost alot more than normal ingestion pyspark pipelines? :(

[@BryanCafferky]

Right. Best use case is for streaming and it has some nice features but it's not for everyone nor is it free. 🙂

[@sumukhds7736]

Hi Bryan, I'm unable to import dlt module using import command I also used magic command and other solutions from stackoverflow too Can you help me to import dlt module

[@BryanCafferky]

Please watch the video. I explain that.

[@ThePrash410]

How to create dlt pipeline using json ?( No option is coming to load json)

[@irfana398]

The worst thing about DLT is you cannot run it cell by cell and check what you are doing.

[@BryanCafferky]

Check this out. An opensource project that lets you test DLT interactively. I have not tried it. github.com/souvik-databricks/dlt-with-debug

[@peterko8871]

I couldn't create the pipeline because it says "The Delta Pipelines feature is not enabled in your workspace." So far I searched for few hours, couldn't find where to set this up. Quite disappointed that your video misses this vital feature.

[@BryanCafferky]

Actually, I do talk about that. See 5:07 where I talk about the Databricks Services. You need to have the Premium service. I did a quick Google search and found this blog to help you stackoverflow.com/questions/71784405/delta-live-tables-feature-missing