How to Perform Great Expectations Data Quality Checks in BigQuery using Airflow!

[@vizartpandey2092]

if you have time, please explain Xcom feature of Airflow !! How one can store the data while transferring to upstream ?

[@thedataguygeorge]

Hey Vizartpandey, have you watched this video? kzbin.info/www/bejne/jXnEnZlumZJsbbs I go into a practical example of how to use Xcom's in it which should help you understand how to transfer data between tasks, but if you think a more strictly xcom focused video would be helpful let me know and I'll produce it!

[@PikachuIzLegit]

how did you manage to install great expectations library into airflow's container ?

[@thedataguygeorge]

Just used the Astro CLI and added it to my requirements.txt file!

[@PikachuIzLegit]

not sure whether it will work for me, will stick to extending airflow image@@thedataguygeorge

[@cdacdbda]

hi Can you please make this setup more detailed ? like how do you setup gx in gcs bucket of cloud composer etc

[@thedataguygeorge]

If you're using Astro you don't need to put gx in a Cloud Composer GCS bucket, or even open source Airflow! Cloud Composers bucket approach is a new unique constraint, which makes installing packages like these harder

[@kienthucnvk8326]

how do you add bigquery_default connection

[@thedataguygeorge]

Go to the Airflow connections menu or add it as an environment variable!