Apache Airflow Architecture 101

Apache Airflow is a popular open-source Linux and Python based workflow engine. Before you jump in and start using it, there are some things you need to know. In this video, I explain the Airflow architecture: the good, the bad, and the ugly, so you can decide where and when to use it.
Join my Patreon Community and Watch this Video without Ads!
www.patreon.com/bePatron?u=63...
Slides
github.com/bcafferky/shared/b...
Video: Don't Use Apache Airflow
• Don't Use Apache Airflow
Follow me on Twitter
@BryanCafferky

The best intro regarding Airflow I found. Thank you...

[@bennage]

I really appreciate the parsimony segment. Very helpful, thanks.

[@shubhamharilalsaroj6380]

Awesome!

[@Learn2UpSkill]

Nice explanation

[@alexchan28]

Hi, great and helpful video, is there a reliability strategy between the scheduler and the dag directory? It seems that's the signal point of the fail point. Thanks!!!

[@BryanCafferky]

Not sure but I think the Airflow UI is where failures are expected to surfaced. But good question.

[@joshi1q2w3e]

How do you feel about “Prefect”? They’re trying to replace Airflow. A video on Prefect from you would be great too!

[@BryanCafferky]

Do you need a WorkFlow engine?

[@joshi1q2w3e]

@@BryanCafferky yes

[@vapornate]

would airflow be an alternative for ssis/ssdt?

[@BryanCafferky]

Hi Nathan, See my other vide to answer that. Video: Don't Use Apache Airflow kzbin.info/www/bejne/j4KTZml7gN92r9k