What's new in Apache Airflow 2.7?

What's new in Apache Airflow 2.7?
🏆 BECOME A PRO WITH AIRFLOW: www.udemy.com/...
Here are the big changes:
➡️ Initialize and delete resources tasks need at ease!
💻 With the setup and teardown tasks (clusters, compute resources, etc)
➡️ Monitor Airflow with the new Cluster Activity page!
💻 Track the most important metrics of your Airflow instance in one click
➡️ New Gantt view!
💻 To better spot task bottlenecks in your DAGs
➡️ New Code view!
💻 Easier to read code changes and verify they've been applied
➡️ New Running and Failed filters!
💻 Click on a button to only see DAG in running or failed states
➡️ New BranchExternalPythonOperator
💻 Choose between tasks while avoiding dependency conflicts
➡️ New cache when calling Variables and Connections
💻 Better optimize resources by reducing the number of calls to the DB
➡️ Limit the concurrency of deferred tasks with Pools
💻 Set include_deferred=True to a pool
➡️ New Graph View!
💻 Navigating your DAGs has never been easier
➡️ db init and db upgrade are deprecated
💻 Use db migrate to create or upgrade the database
➡️ The button for testing connections is disabled
💻 For security reasons. Enable it with AIRFLOW__CORE__TEST_CONNECTION="Enabled"
And more!
🏆 BECOME A PRO: www.udemy.com/...
👍 Smash the like button to become an Airflow Super Hero!
❤️ Subscribe to my channel to become a master of Airflow
🚨 My Patreon: / marclamberti
Ready?
Let's go!

[@donnillorussia]

Please, do some "deep dive" tutorial on managing multiple airflow instances in a cluster like dev/prod. Never used git branching with airflow btw. What are best practices?

[@MarcLamberti]

Will do 🫶

[@JeanBzh]

good point, indeed

[@tturbo3981]

Way to less likes! Your videos are top! Thanks

[@JeanBzh]

setup is just the first task, and teardown is the last task with a "all_done" trigger rule, what do these new labels bring ?

[@MarcLamberti]

Great question. I think the best answer is to look at the following tutorial: docs.astronomer.io/learn/airflow-setup-teardown setup and teardown tasks have specificities that make them much more convenient to manage resources than a task with all_done. I will make a video about it very soon.

[@JeanBzh]

@@MarcLamberti I'll read it right away, thank you Marc

[@mohamedaminchairi]

I'd like to see more about setup and teardown tasks. I'm using airflow intensively but we create our pipelines based on a Yaml file. (custom framework) so our dependecies are taken from the structure of the yaml

[@MarcLamberti]

Will do

[@titowoche]

Please do make a video about setup and teardown

[@MarcLamberti]

Will do 🤓

[@user-xo1rn7re9q]

how to create dags or jobs using this latest ui i dont see any option to do so

[@blind-tr1my]

i am currently have an issue with subtasks in apache airflow. EXAMPLE: let say there are 3 subtasks (in parallerl) which are already completed and now two new tasks are added.. HOw do i make sure that only these two new subtasks are run? (instead of the entire 5 subtasks running) Attempts: from GUI perspective we have option called "queue up new tasks" that does the job. I am coming from the script point of view , about how to activate this button/option Could you please provide me with a hint/idea/suggestions (literally anything )

[@JeanBzh]

I suggest you open a stack overflow topic, I'm not convinced this comment has anything to do with "what's new in Airflow 2.7"

[@user-xy7ed4xq5k]

Where do i find task logs ?

[@MarcLamberti]

On the UI

[@user-xy7ed4xq5k]

@@MarcLamberti earlier there was an option where we can click on task ( task box in the DAg visualization) and go to the logs of particular task. With the UI upgrade how can I navigate to the logs ?