DP-203: 38 - Transforming data with dbt

Hey data engineers!
Databricks notebooks aren't the only way to transform your data. In the latest episode of my free DP-203 course, I discuss dbt - a widely used data transformation solution that offers several advantages over Databricks:
• Simplicity and ease of use
• Data lineage
• Automatically generated and maintained documentation
• Data quality tests
• Jinja templating language
▬▬▬▬▬▬ IMPORTANT LINKS ▬▬▬▬▬▬
My LinkedIn profile: / piotr-tybulewicz-81a8793
GitHub with my drawings: github.com/TybulOnAzure/DP-203
dbt Developer Hub: docs.getdbt.com/
dbt Cloud pricing plans: www.getdbt.com/pricing
Quickstart for dbt Cloud and Databricks: docs.getdbt.com/guides/databr...
▬▬▬▬▬▬ CHAPTERS ▬▬▬▬▬▬
00:00 Introduction
00:28 Where does it fit?
08:28 Demos
45:29 Licensing
46:26 Summary

[@prabhuraghupathi9131]

DBT is completely new to me and I could see glimpses of how much DBT is useful in data transmission, lineage view, testing and for documenting. Thanks Tybul for showing useful tools beyond Azure and Databricks.

[@fekasng2010]

Good one sir.hope iot, event hub and live streaming is coming as well... Have a lovely week ahead

[@TybulOnAzure]

Yes, it's coming

[@maxirojo7829]

Hi Tybul! Thanks for sharing this dbt tutorial. I was not very aware of this tool, I see it very useful to not lose traceability in the datasets and I am interested in the dynamic documentation layer. I have some doubts, the computation used to execute the dbt statements is the databricks computation? And my other question is about dbt and Unity Catalog. If we use Unity Catalog is it also necessary to use dbt? I understand that they are different things but I could see that in a Unity Catalog environment there is a data linage option inside the catalog interface and it is very similar to dbt. Thank you very much Tybul for sharing your knowledge.

[@TybulOnAzure]

dbt is used just to generate your queries, but they are eventually executed on a "data warehouse" which in my case was Databricks SQL Warehouse (a lot of other services/products could also be used: docs.getdbt.com/docs/trusted-adapters). About dbt vs Unity Catalog: both of them have data lineage capabilities so if this is the only feature you care about, then feel free to use any of them.

[@gregt7725]

Świetna seria, dziekuje serdecznie. Jakie elementy kursu należy przyswoić należycie przed aplikacją o pracę(doswiadczenie moje = SQL,PowerBI)? Sam certyfikat zapewne nie wystarczy. Które elementy całego kursu są decydujące przy rozmowie o pracę? Czy Rebrickable API milestone - wystraczy aby kogos przekonac ?

[@TybulOnAzure]

Standardowo: to zależy, np. jeżeli będziesz się starał o robotę związaną ze streamingiem, to Rebrickable API milestone nie wystarczy, bo jeszcze w ogóle nie omawiałem streamingu. Ten pierwszy milestone powinien wystarczyć, jeżeli Twoim zadaniem jako data engineera byłoby głównie zaciąganie danych. Ale nie wystarczy, jeżeli do tego by doszło modelowanie oraz transformacje. Natomiast ktoś mi już pisał, że dostał pracę m.in. dzięki moim filmikom, więc na pewno one pomagają.

[@gregt7725]

@@TybulOnAzure Twoje filmy sa super !! Naprawde 😀- kursy na Udemy sa do bani w porownaniu z tym co tu pokazujesz. Bawie sie ForEach, split, derived columns. A co do modelowania - trenuje Slowly Chagning Dim type 2 na mini bazie danych. Zauwazylem ze sql Stored Prodcedure sa latwiejsze do tego, budowanie tego tylko przez ADF - chrzani sie, jezeli cos ze Source jest usuniete.🤟

[@TybulOnAzure]

Dzięki!

[@TheMapleSight]

If it comes to macros, aren't they the same thing as stored procedures in SQL?

[@TybulOnAzure]

Rather functions

[@pst659]

could you upload daily and finish this series?

[@TybulOnAzure]

No. Yes.

[@TheMapleSight]

Oh, I didn't know that you play Baldur's Gate 3 😅

[@TybulOnAzure]

Yeah, I do. Unfortunately, I don't have as much time for it as I would like to have.