What Is DBT and Why Is It So Popular - Intro To Data Infrastructure Part 3

[@SeattleDataGuy]

If you guys want to learn more about data engineering, then sign up for my newsletter here seattledataguy.substack.com/ or join the discord here discord.gg/2yRJq7Eg3k

[@alexanderlin2022]

If you want to actually know what's DBT, fast forward to 4:20. But, if you already know a little bit DBT and just want to find more advance information, you can skip this whole video.

[@victorberishaj]

Great video, and the Discord sound got me good at 0:57

[@SeattleDataGuy]

I didn’t even notice that hahahah it was a legitimate discord sound

[@michaelperry5442]

Wholly goodness thank you. I've been learning dbt as a Software Engineer and have been trying to figure out why it matters at all. This video helped a ton!

[@RockTrembath]

Thanks! I have been sleeping under a rock for two years - appreciate you helping me get up to speed 🙂

[@SeattleDataGuy]

Happy to help!

[@akashsingh4061]

Awesome Video. 2 years later and DBT has grown even larger.

[@vladimirdinolov852]

It's about time! Excited to watch this.

[@SeattleDataGuy]

Hopefully it exceeds expectations.

[@TheElectrocar]

DBT was a game change for our tableau workbooks. Our tableau workbooks were taking sometimes up to 1-2 minutes to fully load. Once our data engineers added DBT to the mix, the tableau workbooks were taking 3-5 seconds to load. I am not fully sure how all this was done by our data engineers but it was straight up voodoo magic and our end users loved it. Bonus is it made us look good to our 3rd partners who said even their own data team's reports were taking minutes to fully load.

[@bharathraj4545]

I am new to data engineering. I got stuck at the exact same point. Can you help me out and provide any information to contact your data engineering team members?

[@BigQueyrie]

Probably due to the fact that most computations were done in the underlying data sources, and not using the Tableau engine itself. Same can be said about table relationships: aggregating the data to the target granularity helps with that.

[@bharathraj4545]

@jean-baptistequeyrie3388 thanks 😊

[@carldragseth6379]

This​@@BigQueyrie

[@nylarch4227]

dbt is a game changer - it can help create a cultural change where people actually like working with data teams because they see lineage, quality and quick turnarounds built into the system. Highly recommended!!!

[@xXHelsingGamingXx]

SQL and JINJA! 🤯 My Mind is blown!

[@SeattleDataGuy]

😅

[@HVjugo]

We use airflow + DBT in most of our projects now, it's such a good tool

[@SeattleDataGuy]

Awesome. Any tips for future comment readers?

[@tattarrrrattat]

@@SeattleDataGuy I'm from the future...

[@ravishmahajan9314]

So you may be using airflow for orchestrating and dbt for transformation. Can you please tell about airflow? We also have Perfect and Dagster. Do you think they can overtake Airflow in future?

[@HVjugo]

@ravishmahajan9314 I am not experienced enough with prefect to talk about it, so it depends. In our case we were using Cloud Composer, which is gcp's managed Airflow, so it made things easier because the client infra was already hosted at gcp

[@amitesh-rai]

dbt is a great tool. I would really love to see a video from you comparing Prefect, Dagster, and Airflow and how they fit into modern data stack.

[@SeattleDataGuy]

That would be another great video! I have been starting to dig into dagster and prefect a little more.

[@KahanDataSolutions]

Loving your channel and this video. I had a similar start with data pipelines and to me dbt feels like somebody took the frustrations I had with those GUI ETL tools and packaged it up into a better, modern product. Writing code is more fun anyway (IMO). Keep up the great vids!

[@SeattleDataGuy]

Love your channel! I have watched plenty of your dbt videos!

[@yusufs8095]

Great video as always Ben! Thank you for consistently providing us with value.

[@SeattleDataGuy]

Thank you! Hopefully you are doing well.

I was thinking how can achive alone to break into DE field. Then I came across with your letters, youtube and then discord channel. Now I don't feel alone. Amazing contents. Thank you !

[@alexanderpotts8425]

dbt was a breath of fresh air from ssis. I still think ssis was great for its time and still a usable tool in some places. sometimes I miss it... sometimes.

[@SeattleDataGuy]

hahaha sometimes

[@avimehenwal]

very high quality content. Helped me understand some long running doubts I had. Thankyou for sharing

[@SeattleDataGuy]

glad you found it helpful

[@lhxperimental]

Still don't get what is dbt

[@danilomenoli]

It's Jinja with SQL lol

[@TJ34644]

It’s a transformation tool that sucks in data from multiple sources

[@KUBKO17]

Greetings from Vancouver. Thank you for this video on DBT. All other ones were 30min+ :)

[@SeattleDataGuy]

Hope you enjoyed it!

[@datasqlai]

I can correlate with your journey. Started off with using DTS in SQL 2000 and using SSIS first time with SQL server 2005 :)

[@ScottEdwards2000]

same here! always liked DTS better than SSIS anyway. ;-) that said, loving the focus on #SQL that #dbt brought.

[@mahmoudfahmy123]

Elite thumbnail

[@SeattleDataGuy]

Stolen from the DBT slack

[@sumitagrawal5207]

This was great

[@javiereduardochaconarevalo3854]

I always struggle remembering that DBT is not only acronym for Dialectical Behavior Therapy

[@Pegasus1311]

I bet Edwardo

[@MrTeslaX-vi2qn]

good search DBT and it returns that exactly

[@coding3438]

How would dbt replace adf when there to E AND L in dbt? Who’s gonna extract and load data into the warehouse for dbt to be able to transform it?

[@cmaan4life1]

literally just had an interview yesterday where they asked me if i have experience with dbt and i said idk what that is lol

[@SeattleDataGuy]

Now you have a high level idea, hopefully the start data engineering project is helpful

[@henshawefiom5883]

@@SeattleDataGuy can I have your email please for mentoring

[@anildangol]

Great Video man!

[@SeattleDataGuy]

Thank you!

[@Rex_793]

thanks for the video ben!

[@SeattleDataGuy]

Glad you enjoyed it!

[@GlowinginTech]

super helpful content, thank you!

[@SeattleDataGuy]

You're very welcome!

[@davidmorton5949]

“SSIS”. You just brought me back to my 20s 😂

[@roberbonox]

dude how r you, i have the next question, what could i do if i have a stream on snowflake that i want to "consume" in dbt but not creating a physical table or view, instead something live a ephemeral materialization, only to purge the stream and avoid to become stale. I create an ephemeral model and select the stream source but that only create obviously an ephemeral materialization but kind not clean the data on the stream, thoughts??

[@tarunacharya1337]

Nicely Explained - thanks

[@SeattleDataGuy]

Glad it was helpful!

[@metaller_alex]

In my mind, "disadvantage 3" is actually advantage. I follow UNIX philosophy : Do One Thing and Do It Well.

[@marcello4258]

You want to talk about dbt vs dagster? Airflow is kinda nuts using for ETL imo. Airflow isn’t designed for ETL at all - dagster is.

[@MMMS75]

Best. Thumbnail. Ever. Lol

[@SeattleDataGuy]

hahaha, i def. took this meme from somewhere

[@peterg4130]

I'd say SQL is a Pro. If you've worked in BI for any length of time then you know some SQL, so adopted DBT will be very smooth.

[@SeattleDataGuy]

yup!

[@shivabasayyahiremath6802]

@SeattleDataGuy , Thanks for your contribution in help many understand DBT concepts. I have implemented an incremental DBT load(merge) with ProstgreSQL, and the target table is partitioned. The load is performing too good in lower environments and taking half the time it used to take before partitioning. Strangely, when i move the changes to production, its taking double the time it used to before partitioning. Same data volume, Similar DB configuration only difference is, lower env, its server less and prod postgresql is not serverless. any thoughts around this would be greatly appreciated.

[@andreykholkin2737]

Thanks for the explanation!

[@TivoKenevil]

My question is at this point are there TOO Many tools? What do you see in the market; companies using python frameworks (SQLAlchemy, airflow etc) or no code tools like DBT(not considering SQL) more? Edit: i meant SSIS, informática not DBT

[@SeattleDataGuy]

I think we are seeing consolidation. I also think it depends which size of company you are at. I would say Airflow seems to be used more in mid level companies and SMBs it gets harder to manage as you scale. The market always fluctuates in terms of tools. So there is an unavoidable contraction coming.

[@ivani3237]

ETL tools like frameworks in JS... every week came new

[@thedatadoctor]

OMG Ben, I also learned from Wise Owl Tutorials!

[@SeattleDataGuy]

they are solid!

[@Pegasus1311]

Thanks 🦌.

[@aprilleclair6527]

Hi Ben! Your videos are always top-notch and very helpful! I'm wondering what "snp" (?) stands for in this case. dbt would be an awesome tool to learn!

[@SeattleDataGuy]

Glad you are enjoying the videos. I believe I said SMB or small and medium business. Its a classification of business sizes.

[@JimRohn-u8c]

I’m a new Data Engineer and I’m scared to ask what is “Jinja” 😅

[@SeattleDataGuy]

Here is a link pypi.org/project/Jinja2/

[@WallStreetSilver777]

Templating tool so that you abstract away logic with parameters

[@adityanjsg99]

Template language , used in php and Django

[@karangupta_DE]

Hi Ben, do you think no code etl tools like Informatica cloud are way more popular because it's easier to find resources? Or is it just because iics is very mature in terms of implementation.

[@SeattleDataGuy]

What solutions companies pick is a combination of sales, POCs and the head of data or other high level individual's beliefs on whether to go open source, low code/no code, or 100% custom. There will always be a mix of people. Generally all of the various buckets will find their homes.

[@frozenintime]

So it is because its' a watered down ansible to the point someone would only need to know sql?

[@garp9433]

You need more sound proofing in front of you and above you. You can hear surrounding echo or more items in the background or foreground. There's too much echo

[@tanjaa9092]

feedback for your blog - the text colour lacks contrast to the background - makes it very uncomfortable to read

[@yingbozheng4318]

Guess I've been living under a rock then.

[@garp9433]

if this vid was from late 2021 and early 2022 I would have agreed about vc funding, but recently, not so much.

[@SeattleDataGuy]

Totally agree, looking back at the date, this video was April 2022

[@maganzo]

4:23 What is dbt

[@maganzo]

but i still don't know what it is

[@ElaineStafford-u7d]

Armstrong Well

[@sharadov]

Basically Terraform for Data folk.

[@SeattleDataGuy]

something like that

[@thomashass1]

After watching this video, it is still not clear to me what exactly dbt is doing. Examples would have been great.

[@bassett_green]

Okay i downloaded dbt why is my code still awful

[@SeattleDataGuy]

ooof

[@atari1040]

Meltano... Such a disaster... Loved by millions....right..

[@EmersonBruce-c5f]

Riley Ford

[@DonaldKim-w4k]

Langworth Loop

[@MatthewRodger-d2i]

Johnpaul Spring

[@FirstNameLastName-fv4eu]

This guy is the best example when you spend 10 yrs of your professional life in "super-cheap-money-world" what happens, a smart kid with a very vague idea of the real world :)

[@SeattleDataGuy]

You think I am smart shucks. What is the real world to you?

[@FirstNameLastName-fv4eu]

@@SeattleDataGuy explaining the same reason to a Bank where people dont evaluate a technology on "how much money" it has raised. Your generation is just spoiled or scammed by cheap money culture.

[@SeattleDataGuy]

Who do you think is responsible for cheap money culture?

[@xenofongrigoriadis7547]

Is this yet another "platform independence" obsession driven solution? Why should a company, that has decided to go with the Microsoft stack or the Oracle stack or whatever, reject the possibility to use its product specific tools, that give you all the freedom to create your custom ETL processes, while profiting from all specific features of your stack??

[@DianneHamilton-n7v]

Brook Mission

[@MarionFarrar-n7l]

Webster Lake

[@MichaelMick-n1r]

Renee Estate

[@ThomasSlifer-y1f]

Kohler Village

[@SarahHarris-r6y]

Dibbert Terrace

[@JoelFreda-c1d]

Julianne Ramp

[@ClarissaSigrid-p1y]

Augusta Hollow

[@AdelaSampson-e1u]

Bryon Vista

[@pritishukla6433]

Dbt la jazates ka nahi kulthe

[@RandallMoore-q5b]

Hahn Inlet

[@LoganKaloudis-k3w]

Wendy Fort

[@StephenBerg-m9l]

Padberg Junction

[@MickeyPettigrove-m9w]

Wunsch Walks

[@NelsonVic-f1j]

Zieme Crossroad

[@JasonBrown-e4p]

Bauch Walk

[@RicardoDaniel-w3f]

Schmidt Lights

[@RicardoShuey-z1l]

Keebler Shores

[@ThomasFlowers-p1j]

Meghan Turnpike

[@PatriciaClara-r8b]

Anderson Lake

[@SapirHawthorne-v6p]

Jordi Junction

[@EllenJulia-j2l]

Carleton Landing

[@SheridanEudora-k3g]

Cecil Inlet

[@ronaldbarrington5884]

Robel Overpass

[@CharlesMonroe-r8w]

Leanna Neck

[@OscarLillian-d4j]

Dietrich Terrace

[@BlaineModglin-l3d]

Deborah Cove

[@JeromyHensen-e9w]

Cary Overpass

[@ReadeEden-m8w]

Mozell Trafficway

[@BobbyBoris-p3p]

Cruickshank Estate

[@QuillerZebulon-h3f]

Towne Grove

[@CamillaHodge-r9n]

Marielle Rue

[@Overthought1]

*watches first 40 seconds* So let me get this straight: you think most people know what DBT is, but it has only recently been understood that VCs can overvalue startups?

[@JoyceLowery-u1n]

Reagan Forges

[@CecilliaGodfery-x8y]

Olin Mill

[@MorrisBirtwell-s5f]

MacGyver Bypass

[@PearsonBorg-w8f]

Aglae Curve

[@LyndaBartels-c3k]

Huel Junction

[@RalphRich-k9e]

Lesch Spurs

[@DavidBerryman-d6k]

Elouise Springs

[@RayleignFaraday-o5d]

Alfonzo Plaza

[@LawrenceMarkell-r2l]

Wilhelmine Ville