How to Create an ELT Pipeline Using Airflow, Snowflake, and dbt!

In this video, I'll go through how you can create an ELT pipeline using Airflow, Snowflake, and dbt, with cosmos to visualize your dbt workflows! Check out the repo below!
github.com/ast...

[@sainithinreddy6633]

Very clear explanation 👍

[@thedataguygeorge]

Thanks! Glad you enjoyed!

[@maxpatrickoliviermorin2489]

Thank you! Would you mind making a much more elaborate version please?

[@thedataguygeorge]

Sure! What would you like to see?

[@shubhamkawade7351]

Nice explanation! For a specific dag, how do I run a specific dbt command? e.g. How would you execute 'dbt run --select +third_day_avg_cost_run' for this project in the video ?

[@thedataguygeorge]

You could use the cosmos filtering mechanism to filter for just that specific step, but by default cosmos will automatically render each individual dbt model trigger as a task in the DAG

[@WiseSteps.D]

All good, But where we able to see snowflake connection details?

[@thedataguygeorge]

Go to the connection management UI and select the snowflake connection there!

[@Rajdeep6452]

in airflow go to admin>connections and type in your { "account": "-", "warehouse": "", "database": "", "role": "", "insecure_mode": false }

[@ameyajoshi6588]

can we have cyclic pre hook applied for a model? If so how to achieve it using dbt airflow

[@thedataguygeorge]

Yes definitely, if you have the pre-hook applied as part of your dbt model build process it should still work!

[@vijayjoshi-mw8cr]

Hello, I have build a ETL pipeline using python,pandas,airflow,snowflake but problem is when i run the task that time it will not load the data into snowflake.. so please can you help with us!!

[@thedataguygeorge]

What errors are you getting?

[@VijayJoshi-eg2zq]

@@thedataguygeorge when i will run the task that time it will get success message but when check snowflake warehouse i am not able to see the table

[@Rajdeep6452]

So we are supposed to make the dbt init project first and then create the databases in snowflake? And what schema are you using?

[@thedataguygeorge]

The dbt init that's part of the project should create the databases for you as long as you have the proper permissions/setup