Data Pipeline Frameworks: The Dream and the Reality | Beeswax

[@burtgulash]

Airflow is Ruby on Rails and Django of ETL world. It gives you the monolithic batteries included framework which ships with a lot of know how, so that you don't need to reinvent the wheel. The moment you need to do something custom, you're out of luck. Wait for the microframeworks to come.

[@nashdashflash]

It's been > 2 years, what would you say has come or is on the horizon?

[@igrai]

Learned a few things about workflows programming and pain points. Thanks!

[@rembautimes8808]

Good talk very insightful

[@gilbertg.96]

Thanks for the wonderful talk. I'm just getting familiar with data engineering tools, mostly on AWS. Just curious, is AWS step functions ideal in a scenario like this, and what are some perspectives that could be formed towards step functions and data engineering?

[@anonymous4711_]

Pretty cool tech, very good talk. Imagine all this brainpower and engineering didn't go into this surveillance ad system, but to process climate data, research cancer or pandemics. Tragic.

[@matthewprestifilippo7673]

The most interesting thing he said comes at the end where he talks about wrapping airflow