Apache Airflow Architecture 101

[@ivanmaverick1]

I just started to learn Apache Airflow, doing some tests and yes, it's pretty cool, but this intro and considerations in terms of use cases, was eye-opening. The best video I found so far. Thanks for this, Bryan.

[@BryanCafferky]

You may want to look at Dagster too. I really like it. Very focused on data lineage.

The best intro regarding Airflow I found. Thank you...

[@bennage]

I really appreciate the parsimony segment. Very helpful, thanks.

[@alivecoding4995]

How is DataBricks an alternative to Airflow? DataBricks isn't a workflow engine, is it?

[@BryanCafferky]

Yes. Databricks has a workflow service that is powerful. See docs.databricks.com/en/jobs/index.html

[@alexchan28]

Hi, great and helpful video, is there a reliability strategy between the scheduler and the dag directory? It seems that's the signal point of the fail point. Thanks!!!

[@BryanCafferky]

Not sure but I think the Airflow UI is where failures are expected to surfaced. But good question.

[@JimRohn-u8c]

How do you feel about “Prefect”? They’re trying to replace Airflow. A video on Prefect from you would be great too!

[@BryanCafferky]

Do you need a WorkFlow engine?

[@JimRohn-u8c]

@@BryanCafferky yes

[@vapornate]

would airflow be an alternative for ssis/ssdt?

[@BryanCafferky]

Hi Nathan, See my other vide to answer that. Video: Don't Use Apache Airflow kzbin.info/www/bejne/j4KTZml7gN92r9k

[@Learn2UpSkill]

Nice explanation

[@shubhamharilalsaroj6380]

Awesome!

[@De4x4]

Hi guys! I am looking for Apache Aiflow study group. I have some intermediate knowledge on Kafka-KSQL, Spring-Batch, DataEngineering concepts etc. I really want to improove my skills.