Learning Apache Airflow with Python in easy way in 40 Minutes

[@elghs2001]

Seriously, awesome video as I prep for an interview tomorrow morning to refresh/learn about Airflow and Docker. Thank you! The world is so much better with people willing to give their time like you! THANKS

[@tanweer_ali]

Hands down the best video of Airflow and Docker on KZbin.

[@jeffreylim5920]

04:56 - Web UI overview 12:30 - Install Airflow 15:47 - docker-compose.yml 20:47 - build first_dag 27:58 - pass kwarg parameter to the executor 30:36 - second function executor 32:38 - xcom_push

[@SoumilShah]

Thanks man I will add this in comment sections

[@rayees_thurkki]

Which software he is using to write docker yml code

[@ghwhg341]

among all the airflow tutorials on youtube, this is THE ONLY ONE that I can follow and get the exact same result as my life-saver Soumilshah does. others just throw some random and weird errors which made me wanna die. Thank you, life-saver!!!!

[@Кирилл-к2я1ц]

Man, you can't even imagine how helpful for me was your video, thank you

[@jordanfalcao3732]

Thank you very much! I really tried to find a video just with this Airflow basics and all others got errors. This is the first one that run nice and well.

[@jaldipupadhyay4910]

Thank you Soumil, this video has saved me at least weeks of research!!!

[@vedantchaudhari2295]

This video really helps me to understand the basics of airflow very well , thanks

[@christoptimist]

Really helpful for those who want to learn airflow.

[@hamids2065]

It was really nice watching your video :) Good material, and good sense of humor too.

[@abhinandanpatil5027]

Thanks Soumil, this is really helpful to start with airflow.

[@468hitman]

Excellent video, I was struggling to setup airflow from their official site. This made it so easy

[@dhanaraj3361]

Excellent one, Thanks Soumil for all your efforts. It's slightly advanced and it's a great place to have the initial understanding

[@subhendurana6457]

From piles of official docs, you just brought the crisp and clear material for kickstarting! liked and subscribed :)

[@sansri7680]

Excellent video and you made me take off fast in airflow. Appreciate your help friend

[@rarewojak9135]

nice video bro, greetings from Brazil!

[@nikhilghodke3422]

loved ur humility man, nice one

[@enigma_mysterium]

Thanks a lot for your efforts and enthusiasm! Thanks to people like you community expands their horizons!

[@akankshashivhare4573]

It covers very basic but definitely a great one to start with. Thanks

[@kkmahapatra8538]

You sir are very good at explaining minor details. It helped me a lot, so thank you.

[@kaiwang2924]

Wonderful tutorial video. Soumil is talking super fast, which saved us a lot of time learning. Usually it takes about 2 hours, I guess?

[@jamesattwood3454]

Really great video, thanks!!😁

[@demohub]

This video was a great resource. Thanks for the tutelage and your take on it.

[@praveenr4753]

Thank you so much it helped me a lot 😃😃😃😃

[@adrianmromero]

Greetings from Buenos Aires, Argentina, I really learned a lot. I´ll share your video.

[@rahulshukla5033]

Keep going, keep growing.

[@2007selvam]

Excellent One.

[@hoavannguyen7229]

Thanks, Soumil. It helps me a lot!

[@olorundareagnes934]

Awesome tutorial!

[@riteshsharma2585]

Nicely Compiled and presented Soumil !

[@ruthrugezhao862]

Love it and would be eve nicer if you can add timeline to the video and have the command line commands in the description!

[@memez4793]

many thanks for your videon it was very easy to follow:)

[@sachinjain5852]

Hi Soumil, I am working on Tidal Enterprise Scheduler tool now, is Airflow is same as Tidal?

[@awengirr]

Man you hit that Enter HARD! :-D

[@jafarakinfenwa475]

Hi, I followed through the video carefully but I seem to be getting this delay from making my web server come up: "project-webserver-1 | Sun 06 Mar 2022 07:59:39 AM UTC - waiting for Postgres... 9/20 ". after a while, i get this: "postgres:5432 still not reachable, giving up".

[@shivkrishnajaiswal8394]

Helpful video!! Bookmarking it.

[@introvertmind9199]

Hi soumil, Is there way we can store the resulset of postgres read from pghook conmection to txt file and upload it to s3 bucket

[@priyankachauhan3866]

Nice video I found it fruitful 👍

[@munmayabehera454]

Perfect video to set up your environment and start your Airflow journey. Thanks Soumil, that was really helpful.

[@eriktoby483]

sorry to be off topic but does anybody know a tool to log back into an Instagram account? I was dumb forgot the password. I would love any assistance you can give me.

[@noesoren8510]

@Erik Toby Instablaster :)

[@eriktoby483]

@Noe Soren I really appreciate your reply. I got to the site on google and I'm trying it out now. Seems to take a while so I will get back to you later with my results.

[@eriktoby483]

@Noe Soren It did the trick and I now got access to my account again. I am so happy:D Thank you so much, you really help me out :D

[@noesoren8510]

@Erik Toby happy to help xD

[@dragon3602010]

hello, merciii :) is it possible to have the tasks in nodejs for example ??

[@sardormirzaev1265]

That is verz helpful , thanks for your effort

[@birenderdahiya4089]

Well done Man, awesome vedio.

[@ofranceable]

Great intro to this topic Soumil. I did have an error with "start_date" in the DAG(). Solved it by move it to the second param spot.

[@nielshoogeveen3767]

How can I import a local module (created with a setup.py) to use in a DAG?

[@sambidpradhan32]

Hey @soumil , i tried running the script..I am getting import error -ModuleNotFoundError: No module named 'termios'.. i did some googling and found that windows doesnt have termios. I am getting this error while importing python operator. Did you get this error, if yes let me know how you solved this.

[@harikrishnathariboyina6005]

Hi Thanks for good demo. I see you are running Airflow in Windows pc. do we need to install Docker Desktop before we start ?

[@ammadkhan4687]

do I have to build the docker container every time i add new dag?

[@saimanideepanumula9715]

Hey soumil. at 17:43 in the video. I'm getting 'docker-compose' is not recognized as an internal or external command, operable program or batch file. How to fix this in windows 10. I tried installing docker in windows but no use. Please help me with this.

[@Rajgupta-fh3yt]

is ur docker up and running in local ?. if not check ur wsl version,virtulisation seeting enabled.

[@ER-zj3jv]

Great video. Thank you!

[@vinodsamy]

@soumil shah can we create Airflow Rest API using Nodejs as the backend if so can you please how I can config the env for that?Thanks for the great course:)

[@ErikBrakkee]

do you lot need to restart airflow just to add a dag?

[@maxkogan5922]

I'm able to add Dags without restarting airflow takes a few second but it updates automatically

[@MrDaveaneo]

For me, after creating the 3 docker containers, 8080 is listed as a port but is inaccessible. The other two ports work. Looking into it now.

[@BhavanaSamboju]

where did you install the packages and why didn't u create a virtual environment

[@keithwhitson]

yeah, this is a wonderful video and perfect thank you

[@Qzbub]

Thanks a lot bro

[@spradhan36]

Do we need to create multiple python files to create multiple dags. For ex: I've 10 jobs to execute at different time intervals. I want to create 10 DAG's. How to do that ?

[@penugondasaichand692]

how can we get data ?? i mean after doing preprocessing using pandas can we get data through airflow ??

[@manikandankrishnamoorthi1276]

Hi Soumil, where df.to_csv() will be stored ?

[@prafulpatil1800]

Can you please share a video or setup for using Apache Airflow with Databricks.

[@ArounAmitabhDalawat]

Thanks for the video!

[@kenjskjsk]

how to access csv file from my local folder in windows 10? I am okay running bash operator and calling the python script to run but I am having issue accessing local file it gives me error FileNotFoundError: [Errno 2] airflow. Thank you for your help.

[@1UniverseGames]

I have a question, can you use trained machine learning model which create DAG, and integrate it with Airflow or spark, can you make a video on that for job scheduling part.

[@rayees_thurkki]

Which software your using writing code “IJ”

[@soumyadeepmukhopadhyay475]

Hello Soumil, lovely video!

[@rustt1921]

Amazing bro love it

[@abhijitkamune3976]

Awsome video ... Pace is good ...

[@jitrammaharjan9506]

Thank you for this video

[@ParvezAlam-tr6zj]

do we have to repeat the steps everytime we are creating a different project? it seems like we are installing Airflow again and again. Do we have any solution for that?

[@rushic24]

Thanks Soumil, do you know how to install libraries like pip install tensorflow inside airflow container?

[@ayushishukla3138]

Hello sir! How to stop triggering next schedule if previous fail, using wait_for_downstream next schedule is running even though task is not started but i need not to trigger the next schedule at all How to do that sir plz help me.

[@hayathbasha4519]

Hi, We installed postgres image and we didn't saved anything to DB Does airflow store any dag information in db

[@karthikr5884]

Nice video. Thank you

[@sandeshkurhade3242]

sir please show some more code and also how to connacet to pyspark data frame

[@prasadkintali6560]

Nice presentation bro! I have a requirement where we have the schedule interval stored in a table and I need to schedule some python jobs based on the interval fetched from the table. Is it possible to pass the schedule_interval as parameter or is that need to be coded in DAG manually?

[@peteru2872]

you can dynamically set the schedule interval for DAGs using the schedule_interval parameter, but it's typically set during DAG definition and remains static. However, for dynamic scheduling based on data from a table, you can create a DAG that reads the schedule intervals from the table at runtime, using an operator like PythonOperator. Within this operator, fetch the schedule intervals from the table, calculate the next execution time, and dynamically generate DAG runs accordingly. This approach allows for flexible scheduling based on your table data.

[@СергейТравкин-ь1щ]

how to launch air flow? what commands should I enter?

[@imtiazahamed1632]

Hello, I am new at apache airflow. Your videos of airflow are awesome and helped me to understand it. I have a request to you, I don't know is it possible to use airflow for php/mysql simple application Cron task. If yes, then it would be a great help for us if you make a video on it step by step like other videos.

[@SatyaMedidi]

Great Video Soumil. Do you have any sql alchemy or Pyodbc DAG

[@vijaykumar-pt8hb]

Hi, Can I installed Apache airflow using .tar.gz file on production server? Thanks for the video.

[@jinkoofy]

excellent!

[@chowdarreddy3050]

i have tried in the same way but i think we did n't mentioned any role but getting Error " webserver_1 | psycopg2.OperationalError: FATAL: role "airflow" does not exist"

[@ranjithkumarkarthic8696]

Hi Soumil, Thanks for the knowledge sharing. I'm facing an issue while running "docker-compose up --build" command. Help me to fix this error. Below is the full error. ERROR: for webserver Cannot start service webserver: failed to create shim task: OCI runtime create failed: runc creat e failed: unable to start container process: error during container init: error mounting "/run/desktop/mnt/host/c/Ranji th/Learn_Airflow/project/config/airflow.cfg" to rootfs at "/usr/local/airflow/airflow.cfg": mount /run/desktop/mnt/host /c/Ranjith/Learn_Airflow/project/config/airflow.cfg:/usr/local/airflow/airflow.cfg (via /proc/self/fd/14), flags: 0x500 0: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified ho st path exists and is the expected type ERROR: Encountered errors while bringing up the project.

[@akshayshewate6812]

Hi sir can you help with without using docker which one will be helpful debugging application with windows ( VScode or pycharm)

[@akshayshewate6812]

Windows 10 + Ubantu sub system + local airflow server

[@adriannyamanga1580]

damn. Straight to the point.

[@SoumilShah]

Yup I don’t like going round and round

[@NandaAcademies]

DAG means "Directed Acyclic Graph" that should have been told in the beginning of Airflow demo. I think you have missed it!

[@jeffreylim5920]

is "ti" special reserved keyword? 34:23

[@homenature7218]

Hello brother, I try follow your step, but it show error below: (Lerning_Airflow) D:\Mala\Project\Lerning_Airflow\project>docker-compose up --build ERROR: The Compose file '.\docker-compose.yml' is invalid because: services.webserver.volumes 'type' is a required property Could you help to solve it?

[@basavrajningadali8990]

yaml: line 25: did not find expected '-' indicator plshelp

[@diego.camina]

thanks friend! :)

[@shivavarmarebel5134]

Hi I'm facing the dag import error

[@shepherds314]

One suggestion to beat the almighty algorithm. Don't swear in the video. It's not that I have a problem with profanity, but KZbin after it flags you, restricts regions and demotes you in ranking. :( Amazing tutorial

[@vaibhavsikka2982]

hey, awesome video. which coding studio do you use in this video ?

[@ricardopellegrini4975]

Pycharm

[@saurab29]

Hey there, I'm getting an error when I run: docker-compose up --build command: Error response from daemon: failed to initialize logging driver: invalid size: '10ms' Can someone please help me understand it. Thanks :)

[@yeeteshpulstya9890]

Should be 10m?

[@chibuzonwankwo]

Good day Soumil, I hope you saw and red my messages. Please can you do a scratch beginner tutorial on Data engineer and how to install all the necessary tools because in this video I'm completely lost. You didn't show how you installed all the tools you are using for data engineer (i.e. python, airflow, docker and it's container). Please it will be grateful if you do consider. Thanks ☺️

[@venkateshkalyan9429]

Thank you

[@Fatimazahra-lw6qt]

How to install apache airflow in windows please !!

[@RandomGuy-df1oy]

Error Couldn't find env file ...\LOAD_EX=n

[@mathankumars896]

Pls post your linkedin link and blog link in description

[@dharmaputra7394]

How to cicd airflow use gitlab ci?

[@kundansingh-dx9wy]

Sir I have question let say I have use the docker compose --built after that, is it going to create a new image? and if yes is that image is going to contain my new dags automatically?

[@kundansingh-dx9wy]

So if i need to pass this new image to other system do I have need to copy dags there too.

[@grzegorzmajewski591]

Mam dowód że Google nadal używa Apacze 2.0 na Python