Apache Airflow Architecture 101

Рет қаралды 13,649

Күн бұрын

Пікірлер: 17

@ivanmaverick1 5 ай бұрын

I just started to learn Apache Airflow, doing some tests and yes, it's pretty cool, but this intro and considerations in terms of use cases, was eye-opening. The best video I found so far. Thanks for this, Bryan.

@BryanCafferky 5 ай бұрын

You may want to look at Dagster too. I really like it. Very focused on data lineage.

Жыл бұрын

The best intro regarding Airflow I found. Thank you...

@podunkman2709 12 күн бұрын

What really sucks about Airflow is the lack of a GUI for deploying DAGs. Instead, you have to rely on some versioning tool. I don't see any real reason to use Airflow, as everything it offers can be achieved with Python scripts and cron. You could probably write your own "Airflow" over the weekend (and there's a 99% chance it would offer more).

@bennage Жыл бұрын

I really appreciate the parsimony segment. Very helpful, thanks.

@alexchan28 Жыл бұрын

Hi, great and helpful video, is there a reliability strategy between the scheduler and the dag directory? It seems that's the signal point of the fail point. Thanks!!!

@BryanCafferky Жыл бұрын

Not sure but I think the Airflow UI is where failures are expected to surfaced. But good question.

@JimRohn-u8c 2 жыл бұрын

How do you feel about “Prefect”? They’re trying to replace Airflow. A video on Prefect from you would be great too!

@BryanCafferky 2 жыл бұрын

Do you need a WorkFlow engine?

@JimRohn-u8c 2 жыл бұрын

@@BryanCafferky yes

@alivecoding4995 3 ай бұрын

How is DataBricks an alternative to Airflow? DataBricks isn't a workflow engine, is it?

@BryanCafferky 3 ай бұрын

Yes. Databricks has a workflow service that is powerful. See docs.databricks.com/en/jobs/index.html

@vapornate 2 жыл бұрын

would airflow be an alternative for ssis/ssdt?

@BryanCafferky 2 жыл бұрын

Hi Nathan, See my other vide to answer that. Video: Don't Use Apache Airflow kzbin.info/www/bejne/j4KTZml7gN92r9k

@Learn2UpSkill 8 ай бұрын

Nice explanation

@Denilson.marcos 5 ай бұрын

Hi guys! I am looking for Apache Aiflow study group. I have some intermediate knowledge on Kafka-KSQL, Spring-Batch, DataEngineering concepts etc. I really want to improove my skills.