The Realities Of Airflow - The Mistakes New Data Engineers Make Using Apache Airflow

[@SeattleDataGuy]

If you guys want to learn more about data engineering, then sign up for my newsletter here seattledataguy.substack.com/ or join the discord here discord.gg/2yRJq7Eg3k

[@HoopAlchemy]

Underrated video. A data engineer actually willing to give up the secrets ❤

[@make725daily1]

This video sets an exceptional benchmark! -- "Value the journey, for it shapes your path towards unprecedented accomplishments."

[@SeattleDataGuy]

glad you found it helpful!

[@MarcusJFloyd]

Thanks, awesome video. I scaled by deploying Airflow on kubernetes and using the kubernetes executor so that jobs continue to run during deploys. We do have the problem of the dags being in the same repo as our image so running the kubernetes executor was a happy medium. I plan to move the dags to another repo and use a git-sync sidecar container to pull in dag updates at a scheduled interval

[@rembautimes8808]

This is very good content, shares perspective based on specialised work experiences

[@jerbear97]

literally been trying to deploy Airflow in the past 3 days

[@Laikacc]

I am literally trying to install airflow from 5 days

[@SeattleDataGuy]

Have you got it deployed yet! hahah it really can be challenging

[@looklook6075]

@@SeattleDataGuy lol same here. Why so many companies use it. Airflow's design is horrible. looks like it is designed by a bunch a engineers who do not know anything about UI. Something better must come soon.

[@romank7944]

Did you manage to install it, or maybe finally you used another cron-orchestrator?

[@yigidovic]

@@looklook6075 Could not agree more!

[@paul_devos]

I so much trauma from trying to Deploy Airflow 3 separate times at 3 different orgs prior to the "Managed Airflow" era (AWS, Astronomer) that I can't even watch this video. Ultimately, I prefer to work in organizations that are generally smaller, more intimate and greater ownership of their own orchestration locally save for when they have data sets that might be agreed upon to be mission critical at the organizational level and ergo that data set moves to the "hub" where a data mesh like governance system may also take on those data sets in a "hub and spoke" like vibe.

[@neuronqro]

...how about looking at more "modern" alternatives to Airflow? Dagster, Prefect etc. What do you think about their deployment?

[@rnzqt]

Hey Ben, Watched loads of your videos, and in one of your older ones (or a comment afterwards) you mentioned udacity being s good resource for aspiring professionals, but wishing they had a data engineer nanodegree. Now that they do have (a couple of platform specific) de nanodegrees, is it something you have looked at? Potentially my company is willing to fund a course for me, hoping to move into the database from help desk support. Wondering if that would be a good course to get into. Thanks Olie

[@ledinhanhtan]

Because it is bad