How to Create an ELT Pipeline Using Airflow, Snowflake, and dbt!

[@sainithinreddy6633]

Very clear explanation 👍

[@thedataguygeorge]

Thanks! Glad you enjoyed!

[@shubhamkawade7351]

Nice explanation! For a specific dag, how do I run a specific dbt command? e.g. How would you execute 'dbt run --select +third_day_avg_cost_run' for this project in the video ?

[@thedataguygeorge]

You could use the cosmos filtering mechanism to filter for just that specific step, but by default cosmos will automatically render each individual dbt model trigger as a task in the DAG

[@WiseSteps.D]

All good, But where we able to see snowflake connection details?

[@thedataguygeorge]

Go to the connection management UI and select the snowflake connection there!

[@Rajdeep6452]

in airflow go to admin>connections and type in your { "account": "-", "warehouse": "", "database": "", "role": "", "insecure_mode": false }

[@maxpatrickoliviermorin2489]

Thank you! Would you mind making a much more elaborate version please?

[@thedataguygeorge]

Sure! What would you like to see?

[@ameyajoshi6588]

can we have cyclic pre hook applied for a model? If so how to achieve it using dbt airflow

[@thedataguygeorge]

Yes definitely, if you have the pre-hook applied as part of your dbt model build process it should still work!

[@vijayjoshi-mw8cr]

Hello, I have build a ETL pipeline using python,pandas,airflow,snowflake but problem is when i run the task that time it will not load the data into snowflake.. so please can you help with us!!

[@thedataguygeorge]

What errors are you getting?

[@VijayJoshi-eg2zq]

@@thedataguygeorge when i will run the task that time it will get success message but when check snowflake warehouse i am not able to see the table

[@Rajdeep6452]

So we are supposed to make the dbt init project first and then create the databases in snowflake? And what schema are you using?

[@thedataguygeorge]

The dbt init that's part of the project should create the databases for you as long as you have the proper permissions/setup