Airflow Tutorial: Running Data Quality Checks with Snowflake and Soda

Airflow Tutorial: Running Data Quality Checks with Snowflake and Soda
🏆 BECOME A PRO WITH AIRFLOW: www.udemy.com/...
In this project, you will learn:
✅ How to set up an Airflow environment with the Astro CLI
✅ How to set up and configure Snowflake with Airflow
✅ How to load data from an HTTP endpoint into a Snowflake table
✅ How to use python virtual environments to avoid dependency conflicts
✅ How to run data quality checks with Soda and the ExternalPythonOperator
Enjoy ❤️

[@hizokadarkwolf]

I just added this video to my weekend KZbin playlist, but thanks, Mark, for consistently creating great tutorials and sharing tips for data people in simple human language.

[@MarcLamberti]

Thank you ❤️

[@rishirajtandon3849]

@MarcLamberti can we do soda integration wthout storing Snowflake password in code?

[@JoseR-ui9vn]

Do we need Docker Desktop to installed in the machine?

[@MarcLamberti]

Yes

[@orafaelgf]

Great video. This complete environment that you used can I to use in prod? (astro, dbt soda, etc) And eveything is free?

[@MarcLamberti]

Yes except snowflake

[@karantatariya1303]

Kindly assist; Still facing connection issues: getting error snowflake.connector.errors.OperationalError: 250001: 250001: Could not connect to Snowflake backend after 2 attempt(s).Aborting,

[@JoseR-ui9vn]

How to enable Test connection in Airflow.

[@O9W5I7Q]

I have a problem with @task.external_python decorator. It seems that it also requires airflow package in soda_venv virtual environment which is kinda odd..

[@MarcLamberti]

You can find the code in the link in description from this video kzbin.info/www/bejne/eqvbpXaunpmMl6M :)

[@O9W5I7Q]

@@MarcLamberti Particularly, I am getting an error "ModuleNotFoundError: No module named 'airflow'" and "ModuleNotFoundError: No module named 'pendulum'" when running "astro@f867522c2cb4:/usr/local/airflow$ airflow tasks test movie check_movie" (the same error is obtained also if I step into the virtual environment using "astro@f867522c2cb4:/usr/local/airflow$ source soda_venv/bin/activate")

[@O9W5I7Q]

Is this tutorial anywhere on Git?

[@mikekenneth2339]

Great video Marc, Very well put together. 👏👏👏👏👏👏 PS: It would been nice if you add the link to the source movie file in the comment or description for easier follow up.

[@MarcLamberti]

Thank you so much! will do

[@kaycullen8707]

*PromoSM*