Getting Started with Apache Airflow

[@AIEngineeringLife]

Code used for demo : github.com/Sri-nidhi/airflow2.0-demo Puckel docker : github.com/puckel/docker-airflow Airflow Production docker: airflow.apache.org/docs/docker-stack/index.html

[@machinimaaquinix3178]

Srinidhi needs to create her own YT channel! Great questions and explanations at the end.

[@akshaygaikwad4392]

very well explained, thanks shrinidhi for the session.

[@raju5081]

Thanks AIEngineering for arranging the wonderful session. Thanks Srinidhi for that session.

[@rdprasad2225]

Wonderful Session on Apache Airflow.... thanks to Srivatsan and Srinidhi

[@upskillwithchetan]

Thanks for this wonderful session Srinidhi and Srivastan.

[@ravikiranbasutkar1194]

very informative session covering most of Apache Airflow in detail. Thanks Srinidhi n Srivatsan.

[@rajendratiwari5560]

Gr8 presentation by covering most of Airflow aspects in detail. thanks Srivatsan n Srinidhi.

[@shivkj1697]

The question & Answer section was very useful. Thanks

[@centraljerseyresident6302]

Great attempt. So it is alternative approach for traditional scheduling tools like ContromM/AutoSys?

[@techcode_man]

This is a superb explanation and presentation. Thank you srinidhi!

[@sukulmahadik0303]

Good introductory session.thank you

[@gothams1195]

Srinidhi , amazing Session. Just loved it. Need more of these please. Thank you

[@brendoaraujo9110]

Hello, I have an airflow running on my machine with Postgresql on the scheduler's backend and LocalExecutor, but when I put my dags to run it consumes a lot of server CPU, how could I solve this high consumption problem?

[@ashwinkumar5223]

Hi. I want to write a python script to call stored proc and all stored proc as declared in DAG as a operators. How to call all stored proc as one Task

[@biaspind]

Well presented session... Thanks for it. Just wondering when you will presenting next seesion on Airflow.

[@know4good743]

@AIEngineering: can we use shared computer resources from many pc like we were using distribution system resources for the covid19 vaccine super computer with airflow?

[@hayathbasha4519]

Hi is it possible to get task/dag duration from context

[@gopi4321]

@AIEngineering ... I want to create the Dynamic Tasks based on Configuration files ( JSON) .. how to achieve in DAG ?

[@vigneshmurali1994]

Thanks for the session. The PPT fonts are too small to see on a mobile phone. Please present it in fill screen

[@joe_hoeller_chicago]

Is there a git repo for this?

[@AIEngineeringLife]

Here it is - github.com/Sri-nidhi/airflow2.0-demo

[@hitendrachavan6276]

it was a good session but i am from cloud data warehouse background for me to understand apache airflow concept bit tough to understand could you please let me know any other link wherein it will clear the concept from the layman point of view ?

[@AIEngineeringLife]

Airflow documentation by itself is a great one to get started

[@gopinathrajamanickam9475]

Good presentation. only problem I see is the video quality, i believe it is recorded by the host and not the presenter, Please do record from the presenter 's screen so that the blurring due to network speed will not hinder the video quality

[@learnmath9950]

Very good video.. but the video quality was low especially for the code environment.

[@NhatNguyen-bq6jj]

can you try airflow on k8s?

[@AIEngineeringLife]

Nhat.. Are you asking for session on this topic or info?

[@NhatNguyen-bq6jj]

@@AIEngineeringLife Hi, if she can perform deploy airflow on k8s and using Kubernetes Operators, that will be great. I also would like to ask if I use Airflow on docker container, can I use that to process large datasets. Thank you!

[@akshay0725]

Hi, This was a very good session. I have trouble running multiprocess python functions through python_operator. It would be amazing if you could tell me that how to scale a function to use all of my CPU cores

[@arpanghosh3801]

can you please share the presentation

[@AIEngineeringLife]

Added the repo details in comment above which has link to presentation

[@clonlyfans1584]

DAG config is a dictionary not JSON.

[@SMTKMR68]

Time Stemp Please

[@yash1152]

0:47 1:26 freelance data engineer

[@praveena9805]

1:04:25

[@dn9416]

#AIengineering

[@dn9416]

#ApacheAirflow

[@thirutk5401]

What is the total that airflow uses for invoking jvm framework??