Intro To Data Orchestration With Airflow

[@mahtabkhedri6052]

Very useful thanks

[@Astronomer]

Thanks for watching!

[@dataecstasy]

what would be the github integration look like for if companies want to keep the dag code on github

[@Astronomer]

There's a few different approaches, check out this link that goes through various ways you can set up CICD to use github to store your DAG code! docs.astronomer.io/astro/ci-cd-templates/github-actions

[@waterhill]

Can Airflow be used to orchestrate a spark streaming YARN job that pulls data from Kafka and writes to HDFS?.. the idea is if the spark streaming job queues and it can be monitored/alerted/detected and restarted automatically by Airflow?

[@Astronomer]

Oh definitely they can! Check out this link for the different options you have for managing Spark via Airflow, you'll probably want to use a Spark hook registry.astronomer.io/providers/apache-airflow-providers-apache-spark/versions/4.1.5

[@jugsma6676]

If possible, Could you share with us the repo (github) link of Dag's

[@JavierBeneitoBarquero]

This is Marc Lamberti's webinar about sensors: kzbin.info/www/bejne/bnuTmWCMociSapY (you mentioned at 11:30)