Intro to Data Build Tool (dbt) // Create your first project!

[@KahanDataSolutions]

Want to build a reliable, modern data architecture without the mess? Here’s a free checklist to help you → bit.ly/kds-checklist

[@joseantoniolopezmesa6807]

Well I need to say this is great to start on DBT. Not too complicated, not too simple, just something in between. I am watching the whole playlist. I am excited because this will get me up to speed for POC purposes quick enough. Thanks

[@JusmaVids]

I was able to get started with dbt a lot faster than going through dbt documentation and videos.

[@KahanDataSolutions]

Great to hear! Thank you for watching.

[@tatamanasa4398]

@jusmaVids does dbt cloud have an in hand database with data to try few queries on?

[@varinderjitkaur3656]

Hey Mike! Thank you so much for creating this DBT playlist. i am so grateful to you, i had a job assesment exam for Data modelling using DBT and your videos really helped me to learn the basics and later develop my own project. Again thank you so much! 😊😊

[@KahanDataSolutions]

That is awesome! So glad to hear it was helpful you for you. That type of feedback is what makes creating these videos worth it!

[@SnowGateAI]

Great Video For Any Beginner, thank You

[@yanelakantomboxolo185]

Thank you very much. This was the best video for a beginner like myself .... 1 or 2 things that had been changed by dbt since 2020, but I was able to follow you.

[@1ma4ighter]

sweeeeeet! thanks for all the videos, I'mana go through all of these over the next two weeks

[@praveen-dten-0871]

Great tutorial, thanks Kahan for uploading this.

[@parthapratimdas4051]

this is the best tutorial on dbt!

[@KahanDataSolutions]

Thanks Partha!

[@TubeDirektor]

Nicely done, was able to follow through and get my first commit into a snowflake instance.

[@KahanDataSolutions]

Awesome!

[@muchumarriamarnath1858]

It is really good one to start DBT.. Thank you

[@KahanDataSolutions]

You're welcome, thanks for watching.

[@VaibhavPatil-rx7pc]

Excellent information

[@navdeepkumar5946]

Very informative and helpful. Thanks

[@KahanDataSolutions]

Glad it was helpful!

[@v4varun86]

I am getting dbt command not found error in visual studio code. However pip dbt install was successful and its running fine from cmd dbt : The term 'dbt' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again. At line:1 char:1 + dbt debug + ~~~ + CategoryInfo : ObjectNotFound: (dbt:String) [], CommandNotFoundException + FullyQualifiedErrorId : CommandNotFoundException

[@danielejiofor3493]

Please update the video, looks the installation process has changed

[@TalhaIftikhar-wj3bu]

Nice and clear, thank you.

[@mikepain5011]

Super helpful! Thank you!

[@KahanDataSolutions]

Glad it was helpful. Thanks for watching!

[@Ashok-oe3lo]

Thank you very much for your time, very helpful

[@KahanDataSolutions]

Glad you found it helpful!

[@ravibabumca]

Really good Brother

[@pendragonmeat]

Great content. Thanks Mike

[@KahanDataSolutions]

I see you! Thanks bro

[@BobbyWatson-s4z]

10/10 --thanks for the video. New dbt users should view before visiting @dbt. Found this after nearly a day installing/reinstalling to dbt spec (is 'erratic' a good description for dbt docs? maybe...'incorrect'?).

[@parrisgjerde9212]

Nice video. Thanks.

[@ankuraeran1819]

Good tutorial, thank you!

[@KahanDataSolutions]

You're welcome! Thanks for watching

[@miniaturkid]

fantastic playlist. Thank you.

[@KahanDataSolutions]

Thank you!

[@lucymat3746]

so helpful, thank you

[@KahanDataSolutions]

Glad it was helpful!

[@qone2363]

Hey Kahan! There were some changes to get the dbt env setup going.. perhaps it's time for an update? Had to find work arounds.. when I do it, profiles.yml file doesn't exist or cannot be found.. anyways, thanks for making the tutorial

[@khaledtawfik7268]

Hey Mike I love your videos as they helped me a lot. wanted to ask if you have any plan to make a video about incremental techniques and the difference between them and best practices, i always struggle with this as an analytics engineer. Thanks

[@Sunnyb-u8g]

Hi Team , How to see the column lineage and performance for models

[@cherah3012]

Hello @Kahan. Thank you for everything you share with us. It's a pleasure to follow you. For this tutorial, I followed everything well until 11:30. dbt shows me "Completed successfully" => 2 of 2 Ok created view model Public ... etc. But when I look in snowflake under public, nothing has been created ... Do you have an explanation? Thx

[@KahanDataSolutions]

Hello and thank you for your kind words! Regarding your visibility in Snowflake - perhaps you need to switch roles when you sign into Snowflake. For example, if you set your profiles.yml on dbt to use accountadmin , then any role with "less" privileges won't have visibility. This is where access control comes into play, but was unfortunately outside the scope of this video. Let me know if that does it. Thanks again.

[@cherah3012]

Hi​@@KahanDataSolutions Thank you for your quick response In fact, I had done everything right. while I am testing something else in snowflake by creating a table by hand in the same schema (DEMO_DB.PUBLIC, in my case), I saw the table and the view appear in the worksheet (MY_FIRST_DBT_MODEL and MY_SECOND_DBT_MODEL) ... It's a bit strange !! but everything is fine, I will be able to move forward ...

[@stephensun6672]

@@cherah3012 you have to grant privileges to SYSADMIN then you'll be able to see the table.

[@AnkitSinhaforself]

thanks for this !!! really helpful

[@KahanDataSolutions]

Glad it was helpful!

[@juanete69]

Thank you for the tutorial. But the image is very blurry.

[@mayureshpachangane7523]

Do you know how we can write stored-procs in dbt? Can you provide something on that

[@tushargp]

Hello, I have 2 models in same folder , but wants dbt to write out put from each model to separate database in same account, how can I do that? Please advise.

[@KahanDataSolutions]

Hi Dinesh, this can be accomplished by changing the "database" config setting. Here is the documentation: docs.getdbt.com/docs/building-a-dbt-project/building-models/using-custom-databases

[@tushargp]

Thank you, This is what I have been looking for. Can you please advise on below scenario where I have 4 tables to load, each table has 4 scripts to run in specific sequence (insert script + logging scripts), please advise how can I set this 16 script project with dependencies.

[@shankarbonda6977]

@KahanDataSolutions Hi, while running dbt debug, I am facing connection test error, could not connect to snowflake backend, I have populated all the fields popularly. Could you please help me ?

[@0xsuperman]

I don't see the .dbt folder under my admin folder, will this still work?

[@roberbonox]

dude how r you, i have the next question, what could i do if i have a stream on snowflake that i want to "consume" in dbt but not creating a physical table or view, instead something live a ephemeral materialization, only to purge the stream and avoid to become stale. I create an ephemeral model and select the stream source but that only create obviously an ephemeral materialization but kind not clean the data on the stream, thoughts??

[@livu69]

Great tutorial!

[@KahanDataSolutions]

Thank you!

[@n_128]

The command "pip install dbt" will not work anymore. The documentation says: "Note that, as of v1.0.0, pip install dbt is no longer supported and will raise an explicit error. Since v0.13, the PyPi package named dbt was a simple "pass-through" of dbt-core and the four original database adapter plugins. For v1, we formalized that split." This command emulates the behaviour of the pip install dbt command: ``` pip install \ dbt-core \ dbt-postgres \ dbt-redshift \ dbt-snowflake \ dbt-bigquery ```

[@KahanDataSolutions]

You are correct - This video is now a bit dated compared to the latest version but if you make the adjustment you mention you should be all set. Here is a reference link for others who are interested: docs.getdbt.com/dbt-cli/install/pip#pip-install-dbt

[@ShreyasSureshDhamore]

time stamp 7.38 in this vide I have see when you do dbt init then you can see which db you need to be configured instead of doing manually.?

[@mounikarepakula1910]

Hey Mike! Thank You so much. May i know how to create a new table in DBT tool?

[@KahanDataSolutions]

Thanks for watching! To create a table you create a model (sql file) and set the materialization as "table". You can follow along the rest of the videos in this playlist to learn more about how to use dbt.

[@StephenRayner]

Please make videos on “Meltano” and how to use dbt within this? 🎉

[@jaredfiacco1107]

Thanks Kahan!

[@KahanDataSolutions]

You're welcome!

[@SebastianFeriz]

No se que pasa que al querer hacer el init no reconoce el dbt... busque para agregar el path y no hay caso.. Tambien probe instalar el dbt-core y tambien hacerlo via virenv... y no pude tampoco... A alguien le ha pasado??

[@sungod595]

Hey Mike, Excellent Content. Many Thanks. I know these videos are good to start with . Do you plan to include any videos on advanced concepts Once again Thanks so much for great content

[@tushargp]

Also if you can please advise how can I run update statement.

[@bashabash3697]

hey Mike. Can we install DBT in Azure data factory

[@udaykumar-tb5kn]

Dbt is cloud login right why did you install it locally?? What difference between on prem vs cloud dbt login pls help

[@tushargp]

Hello, Please advise how can i run update statement on snowflake.

[@sakthivadivel1057]

Hi @kahan, While I try to install dbt (pip install dbt) an error occurs.Error:KeyError:ICU_VERSION.could you help me to fix?

[@KahanDataSolutions]

Hi there - Unfortunately that sounds like an environment issue specific to your machine/setup and not something I can easily answer. Make sure you are using an acceptable version of Python for dbt and try searching around online to see if others have encountered similar issues.

[@vishwanathmuthuraman5530]

I was able to pip install dbt, but when i run " dbt init [project name] " it says something like cant recognize path, external, or batch file ?? what do it do ?

[@Faith_Ojumah]

HI, I am having difficulties finding my profiles.yml file on my MacOS. Please help?

[@tatamanasa4398]

does dbt cloud have an in hand data base with data to try out the commands?

[@KahanDataSolutions]

Hi Tata - Unfortunately DBT does not come out of the box with a database, but instead is designed to plug into one that you already have.

[@blancamola]

Hello there! I got stuck at the start. I installed dbt by pip with no errors (Actually dbt-redshift). But when I run dbt --version or any other dbt command I get the typical error of not recognized as internal. I tried to add .....\python310\site-packages in the path of environment variables, but no luck. Thank you for any help, Looking forward to watching the rest of videos too. 😃

[@samuelbrown797]

Hi Serna. Sorry for reaching out to you but I am encountering the same problem. Were you able to find a solution to this problem?

[@shrero]

Will this intro work on a mac as well?

[@sararodriguez5803]

Can you possibly configure dbt profile to work with Spark into AWS S3 parquet files?

[@KahanDataSolutions]

Here are the list of available adapters - docs.getdbt.com/docs/available-adapters

[@anjanashetty482]

How to update records in dbt. Update operation is failing upon dbt run.

[@PawanKumar-un7sk]

After the command applied pip install dbt When I check for dbt --version it's showing that dbt is not recognized as an internal or external command, operable program or batch file what I do now?

[@KahanDataSolutions]

Did the pip install dbt command run successfully?

[@lokendrasinghtanwar5917]

can you please make a video on how to setup dbt in VS code

[@jeremiahworkman1631]

Is this tutorial applicable (after the initial install for WIndows) for MacOS?

[@KahanDataSolutions]

Hey Jeremiah - Yes the general workflow of dbt will be the same across operating systems.

[@frankbell3033]

when I run your dbt init step in the beginning on a mac I keep getting an error raise ValidationError(f"Unable to create schema for '{type_name}'") hologram.ValidationError: Unable to create schema for 'Optional'

[@mechy6065]

Hello Frank, If you are running Python 3.9, One of their dependencies is not compatible. You will have to downgrade your python version maybe 3.8. If you had found any other solution please reply to this message.

[@CosmicNomad]

This tutorial is good, but unfortunately the installation/ project creation steps are outdated. The dbt installation process has changed, the dbt init now asks adapter credentials before it can create a profiles file etc. :( Anyone using this video would have to do a lot of research trying to simply set up their first project.

[@grantdoyle1259]

Ive done everything correct up to this point but when I run dbt debug I get an error "dbt : The term 'dbt' is not recognized as the name of a cmdlet, function, script file, or operable program."

[@samuelbrown797]

Hi Grant - Sorry for reaching out to you but I am encountering the same problem. Were you able to find a solution to this error? Thanks Sam

[@grantdoyle1259]

@@samuelbrown797 I ended up just working with DBT in a virtual environment. They have some documentation on it and I believe he has a video on it. I am taking a DBT udemy course that has been solid in setting up and configuring DBT

[@mohitsondhi]

where do i find profiles.yml file when installed in mac?

[@KahanDataSolutions]

It will be on your home directory at "~/.dbt" However, it might be hidden by default in the finder so you need to press "CMD + Shift + ." to show hidden directories.

[@mohitsondhi]

@@KahanDataSolutions Great, Thank you so much, I am about to buy your course, have 1-2 questions on that, can we somehow chat or have a call?

[@litttlemooncream5049]

hey mike. Thanks for your video. really helpful! I wonder if I can use python to do loops instead of writing 1000+ rows sql? is it possible to combine dbt and python

[@KahanDataSolutions]

I recommend checking out Jinja, which allows you to incorporate more dynamic things like loops into your SQL code. docs.getdbt.com/docs/building-a-dbt-project/jinja-macros

[@ademolaoladapo9024]

Good Morning, when i tried checking the version of dbt, i get this error - 'dbt' is not recognized as an internal or external command, operable program or batch file. How do i proceed?

[@KahanDataSolutions]

Did you successfully install dbt before running that command? Perhaps you need to open a new/fresh terminal.

[@ademolaoladapo9024]

@@KahanDataSolutions i did install dbt using pip3 before running that command

[@hantt]

This means that dbt is not added to the path of your execution. Make sure when you run pip install that your virtual environment is active.

[@MrEsdrasadiel]

Hello! I am watching your video, for now the dbt request to choose an version in according the situation. In my case i choose the dbt-redshift to install. "pip install dbt-redshift"

[@KahanDataSolutions]

That's correct! Here's more docs on Redshift specifically: docs.getdbt.com/reference/warehouse-setups/redshift-setup

[@SravaniK-ee8td]

How to avoid hardcoding of username and password?

[@TheSimpGatsby]

thanks mate.

[@KahanDataSolutions]

Thanks for watching!

[@prateekganguly9285]

If i were 3 years younger I'd have pushed the code with all the secrets in the yaml file

[@xhshaw4930]

nice👍

[@KahanDataSolutions]

Thanks!

[@jaym7178]

Hi I am looking into a career as a Data Engineer i am overwhelmed with software I should learn other than learning python,sql is the any tools and areas I should focus Thanks

[@AbdullahShoukat-h5c]

thanks!

[@vickysworld9]

I am unable to install dbt on windows from CLI "Error: Could not build wheels for cryptography which use PEP 517 and cannot be installed directly"

[@KahanDataSolutions]

What version of Windows and Python are you using?

[@vickysworld9]

@@KahanDataSolutions windows 10 & python 3.9

[@vickysworld9]

@@KahanDataSolutions Please help

[@KahanDataSolutions]

stackoverflow.com/questions/59441794/error-could-not-build-wheels-for-cryptography-which-use-pep-517-and-cannot-be-i

[@graceluo9813]

@@vickysworld9 found that python 3.9 does not support dbt's snowflake connector hence downgraded my python installation version to 3.8 which helped

[@atulbharti9757]

if your DBT INIT NOT WORKING PLEASE TRY THIS ONE python -m pip install --upgrade dbt-core. pip dbt install not working please try this one pip install dbt-core

[@abdullahsiddique7787]

How’s dbt as a career with snowflake

[@KahanDataSolutions]

There is a big demand right now for dbt developers, particularly with Snowflake. Having said that, I would consider dbt as just one of the many possible tools in the broader career of a data/analytics engineer. Either way - I'd strongly suggest learning it!

[@abdullahsiddique7787]

@@KahanDataSolutions thanks bro for replying , so if we learn dbt is there any need to learn ETL tools like informatica talend etc and is your playlist complete course on dbt

[@TJ-hs1qm]

The terminal on Windows is an abomination, that shouldn't exist 😭

[@emanueol]

nice playlist, but i found really annying KZbinrs not buying decent microphones and forcing viewers to hear every single mouse click and every single keyboard key press.. sorry for being honest, but we can only improve with others suggestions

[@KahanDataSolutions]

Appreciate the feedback & glad to hear the playlist was helpful! I've also upgraded my microphone since this video and you can see it on my newer videos.

[@InPersian]

Unfortunately, I tried to install dbt but it is not working"\Python311\Lib\site-packages\mashumaro\meta\helpers.py", line 161, in is_generic raise NotImplementedError, I tried to find the solution but still no result

[@sateeshbabu5792]

Hi , This is sateesh, i have completed snowflake snow pro certification. planning to learn DBT. What are the pre-requisites ? and complete Python is required before learn DBT ? Could you please let me know.

[@KahanDataSolutions]

I recommend having a solid understanding of SQL as the #1 pre-req. You don't need to be a master Python dev to use it. Very beginner understanding of general programming is all you need to get started.

[@sateeshbabu5792]

@@KahanDataSolutions Thank you so much for your reply. Are you providing online training on DBT ?

[@KahanDataSolutions]

@@sateeshbabu5792 I do have a dbt course. You can check it out here - www.kahandatasolutions.com/the-playbook-for-dbt