Airflow for Beginners - Run Spotify ETL Job in 15 minutes!

[@ElPapelMan]

One of the best etl series I've ever watched on youtube... thank you.

[@mehdiyahiacherif2326]

well , i had a BI (business intelligence) project this year and i had no idea what are those etl and reporting tools , i searched for like 10 days and tested a lot of softwares some of them were usefull and some of them were ... meh and i actually liked airflow and dbeaver and both are in this video what a surprise , for people who wants to test some bi tools tou have (free and open source) : etl :airflow , knime , pentaho DI reporting : superset,also some cool dashboards in pentaho server db RAT and gui tools : dbeaver and also DbSchema data mining : tanagra and weka take a look also at apache kylin (i did not knew how to setup it to get postgres as a datasource so ...) good luck guys and great video lady

[@karolinasowinska]

Awesome, thanks so much for the tips! :)

[@mehdiyahiacherif2326]

@@karolinasowinska thanks for your reply , I would be very happy if you make a video of how to connect apache kylin to nex data sources ( it is painfull lol I searchedand itis not well documented) A serie about bi tools and data manipulation can be a great idea since not a lot of ppl do it on youtube Good luck

[@ashishk81]

i am data scientist by profession and wanted to learn data engineering in details , i didnt found single free online resource to learn all data engineering skills ..you are doing great job ..waitng for your videos

[@karolinasowinska]

I'm super glad my videos are useful! :)

[@ashishk81]

@@karolinasowinska Can you please suggest free online resources to learn end to end data engineer

[@tejagoud4871]

Watching this is really worth the time. Not like other KZbin channels where they run promotions for a minute or two. Above all, it is really a good video on getting started with Airflow. Great work Karolina. You are an amazing instructor.

[@karolinasowinska]

I really appreciate this comment, thanks so much Teja!

[@ricardoarbois2839]

@@karolinasowinska Nice video.i really love your installation., hope you dont mind if i post here my yt vid about installation airflow in heroku....thanks and please more vid in airflow...kzbin.info/www/bejne/jpKXg6pjmMd-f7M

[@itsrainingcatsanddogs]

I'll be needing more of these airflow tutorial

[@karolinasowinska]

I'll try to do my best! :)

[@quinnluong114]

She a real one, you can tell because she's showing all the problems she's running into

[@avinandanbanerjee9568]

Important note - You won't find the airflow directory until you run something on the CLI using airflow Just type in airflow once and hit enter to find the config file

[@kelvin5685]

Thank you! Also one needs to run "source airflow-venv/bin/activate" before running the command "airflow". That way you don't get an error that "airflow" command is not found

[@jae11011]

Great video for airflow beginner!! I have tried to run airflow for too much time and always stuck even before start webserver. This is the first time I successfully run it!! For anyone who is also new with Airflow, I got some small issue when I follow with the video. Here is how I solve it, just in case anyone encounter the same issue. To start airflow 1. After install airflow, need to run airflow first, to create the airflow.cfg in the home.(If you haven't run it before) Simply type "airflow" will do the work. I didn't run it first, so I couldn't find the cfg file anywhere. 2. I also need to run "airflow db init" to create the db for logs. 3. Last, I need to create a user before I use the webserver, otherwise there will no user for me to login. These steps are available in the airflow document quick start as well. To run the dag as in the video. 4. I switched the toggle to ON in the dag view, otherwise the task will remain in running forever. 5. To run the extract.py or run_spotify.py, I need to put extract.py in the dags folder first. I just put the file directly in the dag folder, but I saw others put the whole python package(subfolder with __init.py) in the dag folder. The latter approach is better for bigger project with no doubt. But I still want to know does everyone put the packages directly in dag folder in real world? Since it's still a little messy for me putting dags file with scripts itself. A few questions I have though, should I terminate and restart airflow scheduler everytime I change my script or it will pick up automatically? I am still having token expired issue when run the script in airflow, even I updated my token in the script and ran fine in local machine. But it is a awesome video to me! Thanks to Karolina!

[@Indianvloggerinfinland]

Hi Karolina , Thank you for the video. I need a help as am kinda stuck at one place where you will be editing the airflow.cfg file. I use Macbook. I couldnt find the file even after installing the aiflow. I dont see the folder airflow at all in spite of giving the command "export AIRFLOW_HOME=/airflow" . Need your help on this.

[@SASUKEUCHIHA-yc6er]

The moment I saw the video Thought she has over a million subscribers She deserves more subscribers and also more views....

[@karolinasowinska]

Oh, that's very nice to hear, thanks! :) 🙏

[@ChernobylPizza]

I like your tutorials because they are simple. I kinda got stuck biting off more than I can chew and I was going in circles for a while. Data careers are about so many things it's easy to get lost (python, hadoop/spark, airflow, ML, math/stats, visualization, cloud).... I just needed something simple I can do easily to get going.

[@romanlukichev4971]

This is the first time I've heard about Airflow. Apache Airflow is an open-source workflow management platform for data engineering pipelines. It started at Airbnb in October 2014[2] as a solution to manage the company's increasingly complex workflows. Creating Airflow allowed Airbnb to programmatically author and schedule their workflows and monitor them via the built-in Airflow user interface.[3][4] From the beginning, the project was made open source, becoming an Apache Incubator project in March 2016 and a top-level Apache Software Foundation project in January 2019. Airflow is written in Python, and workflows are created via Python scripts.

[@rodrigoamoedo8523]

your content is getting better every time

[@karolinasowinska]

Oh, really? I'm super glad you think so! :)

[@geetanshkumar1854]

Amazing work ma'am. I am new to all this and this tutorial was so simple and clear. Your way of explaining is also unique because you talk about errors as well which very few people do.

[@karolinasowinska]

Thank you, I'm glad you like my talking style! :)

[@buithanhlam3726]

I found it very hard in the documentation, the book... then I found your video. Thanks a lot!

[@samuelabolo]

"back to downgrading our future" got me cracked up

[@Marc_B.4]

The video I was waiting for! I'm happy to see it, it's very well presented. It was really useful, I now have a good feeling about how Airflow works. Can't wait to see what's next on your channel :)

[@karolinasowinska]

Aw I'm super glad to hear that it met your expectations! Thanks! :)

[@henriviss]

Karolina, I was enlightened by your explanation/methodology, helped me a lot to get started with Apache Airflow, mad props for this! Keep up with the work! Cheers from Brazil

[@karolinasowinska]

I'm so glad that my effort didn't go to waste! Thanks for your comment! :)

[@TaylorNelson1]

Ah the joys of finding new errors when you try to install things for a new production environment... this is such an accurate depiction of real engineering life.

[@karolinasowinska]

Exactly! ;)

[@imdadood5705]

Something that I was looking for. I know Python, SQL, R and a good amount of machine learning. But I didn’t know what to do next. I just search for Apache Airflow and I got this! Thank you!

[@AlexAcostaB]

This is such a great introduction to Airflow. I already designed one pipeline and I am ready to implement it. Thank you so much.

[@karolinasowinska]

That's fantastic, how did it go? :)

[@AlexAcostaB]

@@karolinasowinska it work well. I’m getting data from one customer’s FTP and it is failing using Python 🐍 working on a solution and it will be ready for deployment

[@karolinasowinska]

@@AlexAcostaB Awesome stuff :)

[@shivkj1697]

@10:16 why should we avoid passing data to operator/Task from its predecessor operator? Passing data enables creating a dynamic pipeline

[@marvhan888]

what a well done mentor's job you are doing.

[@karolinasowinska]

Thank you!

[@Paperwood360]

What a great tutorial, the best I've seen so far for Airflow! Thank you very much

[@josecarlossilva3670]

Awesome content!! I ve been struggling with that stuff for few months. Thanks for sharing

[@Shagysami]

Thank you for the neat video ! I'm new to data engineering and may be nailing my upcoming job interview thanks to you

[@karolinasowinska]

Wow fingers crossed!;)

[@adityagaikwad3172]

Great video, short and to the point. But, I was wondering if the job executes daily, wouldn't our token expire. Maybe we have to update it manually.

[@karolinasowinska]

You're 100% right, it would. But there is a way to automatically download the token and have that in our script, so that it is fresh every time the program runs:)

[@adityagaikwad3172]

@@karolinasowinska Thanks, I'm using authorization code flow (OUTH 2.0) from the Spotify website. Where I had to manually get the auth code every couple of hours. I'll try your method.

[@baotran4175]

Thanks very much. I come from Vietnam. Right now intern Data Engineer. I hope you can do more topics on Data Engineers in the near future

[@karolinasowinska]

Hello there! Nice to hear from a fellow techie. I will do for sure! ;)

[@thepakcolapcar]

Great video. Is there a way to pass configurations to the DAG and also they can be accessed by different tasks with in the dag? I am aware of XCOM and Variables etc. But is there a way a config file in form of json or yaml can be passed to the dag? And without using xcomm or variables from admin menu is there any other way to set and get values across diffetnt tasks with in dag?

[@NewyJimmy]

More on airflow please! This was great!

[@karolinasowinska]

I'll see what I can do! I'm glad you enjoyed it! ;)

[@pranoygowda4595]

Could get on the concepts and working with Airflow just by watching the video. Very much helpful video to get started with. Amazing!

[@karolinasowinska]

Glad you enjoyed it!

[@blakegirardot5813]

I had to run `airflow --help` after installing or the airflow folder and .cfg file were never created. Any airflow command should trigger the generation of the expected directory and files.

[@McMurchie]

Great video Karolina, for those struggling with pip install - I suggest doing a quick learn of conda so you can create a quick conda environment to install airflow without messing up your primary python/pip libs and versions. I agree though, Airflow is so tricky to set up.

[@mahammadnabizade9408]

Thanks for the amazing tutorials, just curious why you created a different virtual env for airflow ?

[@randolphralph8322]

This is a great tutorial. I am having difficulties setting this up in Windows 10 environment. I was able to setup the virtual environment, but the install process for Airflow differs.

[@nskeip]

Nice video. And about direc acyclic graphs - actualy, you could draw an arrow from 3 to 2 in the graph you showed as an example ^_^ (because there was no way to go back from 2 to 3, so it would not make a cycle)

[@brendoaraujo9110]

Hello, I have an airflow running on my machine with Postgresql on the scheduler's backend and LocalExecutor, but when I put my dags to run it consumes a lot of server CPU, how could I solve this high consumption problem?

[@nataindata]

Karolina, thank you a lot for your efforts and for making these videos! You've sparkled ingenuine interest in me to try the project out. Plus, it's really great to know that Data Engineering community is empowered by women. I'm only starting my way in DE, so it's great to follow you and learn. Love ❤️

[@karolinasowinska]

Thanks so much for this lovely comment! Good luck on your DE journey and I hope I'll see you around here! :)

[@solagan7402]

Hello, you agree that the laptop where you configure airflow etc.. needs to be on to run the etl daily ? If the laptop is off nothing will happen right ?

[@MrDavisv]

Best explanation I’ve see about DAGs and super helpful intro to Airflow. Makes complete sense. Thank you!

[@karolinasowinska]

I'm glad this was helpful! :)

[@sirosala]

Excellent Karo !!!! 💪💪💪

[@souravsingh6188]

hello karolina, i am just college grad and wann learn how to start carrier in data analysis

[@josesebastiancolaneri7125]

Excellent video! Thank you very much Karolina!

[@pantepember]

3:00 In my case, I encountered a different error and it required upgrading pip to its latest version and adding include-system-site-packages = true into the pyvenv.cfg file.

[@softwaresteve]

How long is the DAG actually running for? I followed the code and my DAG ends up running in an endless loop. I copied the code from you github and it also runs in an endless loop. What have you done differently that isnt in the video? The Github code is missing an import of 'from datetime import datetime' as well. Is anyone else running into this issue?

[@softwaresteve]

For anyone else running into this issue, you will need to activate the DAG in the airflow admin panel. There is a switch to turn the DAG on that will complete the dag.

[@sidhusam]

yes, I'm facing the same issue. DAG is running state forever

[@sidhusam]

@@softwaresteve finally able to resolve this issue thanks

[@DDAN48LIFE]

I love Karolina , you are the best

[@McCallumClips]

Your video was exactly what you said it would be. An introduction. VERY GOOD JOB! Thank you.

[@karolinasowinska]

Yes indeed, it'd be very hard to discuss details in a 15-minute video! I'm glad you liked it! :)

[@dalicodes]

Nice video thanks. But 6:45 you can draw an arrow from 3 to 2 and it will still be a dag

[@hisky74]

The nano tip is very useful!! Very good content! Thank you!

[@karolinasowinska]

Glad it was helpful!

[@karolinasowinska]

If you'd like to learn data engineering, I recommend following the 4 simple steps below to land you the first job interview: 1. Learn Python I recommend following the Python for Everybody specialization course on Coursera, which is one of the most popular courses there: imp.i384100.net/x9gVO3 2. Learn SQL SQL is still the lingua franca of data. I recommend going with Learn SQL Basics for Data Science course, because it contains some chapters which are very releavant to data engineering in partcular, e.g. distributed computing with Spark imp.i384100.net/QOMZ09 3. Learn Bash scripting/Linux I wouldn't take a full course on it, but at least read a good article. if you do prefer to take a course/guided project, I think this one is short and good: www.coursera.org/projects/command-line-linux 4. Learn how to develop on the cloud, e.g. on AWS There are a few good courses around there, but I think the Coursera one is the most comprehensive imp.i384100.net/P0MJBM

[@PedroCarneiroJr1971]

Hey, Karolina! Please check the Coursera link at item 4 because it's returning "bad merchant".

[@zma314125]

Thank you for explaining such a complicated topic in a simple way. This will definitely be a help as a foundation to data engineering. Keep up the great work!

[@saurabhkhanolkar5017]

Hey! Thanks for this... One question though : How does one implement the same thing in AWS?

[@karolinasowinska]

That's a topic for another video! :)

[@saurabhkhanolkar5017]

@@karolinasowinska Hey... It'll be great if you could make a video on this !

[@chrish.4734]

Great video, thanks a lot Karolina! I really like your clear way to explain, which is straight to the point and your great energy!

[@heikokraemer2735]

Thank you Karolina, very useful, totally no waste of time.

[@karolinasowinska]

I'm glad it was useful! ;)

[@saad1732]

Beginner view on deploying managed airflow ETL task on AWS? Also, AWS airflow vs glue?

[@alexanderbenavides1887]

Wuau, an amazing video tutorial. I love your videos :)

[@AndresHernandez-mz3xh]

Wow, thank you so much Karolina! It helped me a lot with my project!

[@karolinasowinska]

My pleasure!

[@OPopoola]

Thanks for this introduction. I have been wondering what the big deal is with Airflow. Now I see the potential.

[@karolinasowinska]

My pleasure! :)

[@Nedwin]

Love the extract, transform, load part! ❤️

[@JesseHughson]

You can actually connect 3 to 2 in the example graph. This would not create a circular dependency since the relationships are directional and there's still no way to go backwards. You could not connect 4 to 1 or 5 to 1 however, as this would create a circular dependency.

[@troymann5115]

Nice video! One thing to think about concerning running Docker containers from the DAG: Airflow 1.x apparently has an issue which leaves containers in a non-started state. (At least it was a problem in our environment.) Airflow 2.0 seems to have resolved it. Thank you for making this video.

[@edragon1412]

This is really helpful for Airflow beginners like I am. I am appreciated your work a lot. Keep working those topic like this, girl ;)

[@karolinasowinska]

I'm glad you enjoyed it! I will keep it up ;)

[@sanchesrfl]

DAG = a directed collection of tasks without going back. Thanks!!!!!!!!!

[@karolinasowinska]

You can remember it this way too :)!

[@iitian2012]

There are so many tools in the market for data extraction and loading.Which one to choose?

[@snehajain7946]

More on airflow? Any new video? Also if you could also do a tutorial on LUIGI...we could compare easily which one to chose from

[@shadowvectors]

i didnt get the airflow file in ~/airflow path after installing apache-airflow using constraints, but when I ran airflow db init I got the files in the path of airflow, is correct if I am doing it like this or it should automatically create files?

[@pushpanthkumar9028]

Would like to know why xcomm variables are not recommended

[@akbarazad93]

Thanks Karolina! I have an issue when replicating this pipeline's framework. It's regards to the ETL part. I was able to run the DAG successfully and when I went into my sqlite database, the table was not updated (no new records appended) from the DAG run. Any idea what could be the issue? I'm using Python 3, Ubuntu on Windows OS. Thanks!

[@PenStab]

Really impressive explanations and teaching approach. You were concise but covered so many small in-between points that I would have otherwise missed. I'm definitely subscribing and going to watch other videos! My only complaint would be the resolution of the capture of the VS Code window - can it be a 16:9 ratio? It was so small on my phone.

[@karolinasowinska]

Thanks! I'll try to improve resolution going forward ;)

[@medotop330]

What about apache spark ?? I am waiting to see in your tutorials about it Thanks too much : )

[@karolinasowinska]

I'll see what I can do!;)

[@fajarabdulkarim7672]

Hi carolina, thanks for nice video. Easy to understand. Btw do you know how to test the dag ? Its Like unit test or the test which QA did in software development

[@nachoggz]

Excellent video! Someone know how can I import a postgres certificate for my db connection? Im trying but airflow can not find the file. im running airflow on docker

[@pushpanthkumar9028]

Do we know what happens if the start date is not hard-coded?

[@yuriershov6530]

Great content! Just what I needed before starting my data engineering courses

[@karolinasowinska]

Glad it was useful! :)

[@GustavoLeig]

Thanks for this awesome tutorial Karolina, one question, the first time I run, the Airflow starts lots of jobs, and the final table gets 20 songs, is that correct? Why so many jobs to get 20 rows?

[@Abhi-gq6zx]

Ma'am, please share some learning course on Python related toData engineer.

[@karolinasowinska]

Certainly! I'd actually soon be releasing my own course on how to enter the data career. If you'd like to get alerted, feel free to drop your email address :)

[@nardove]

Thank you for the videos, your short tutorials are great, the only downside (for me) is your coding window, the text is too small.

[@karolinasowinska]

Thanks for the tip!

[@BestevertechBlogspot]

Thank you for posting such relevant content. These are really worth it!

[@karolinasowinska]

My pleasure! :)

[@fedemoralesrampulla4332]

Excellent video!!! Was great but cannot pass the "install" part... try 1, 2, 3 times and I couldnt.

[@smrutiranjanmohapatra7691]

Can you make video on mwaa(managed work flow for apache airflow) in AWS??? Like for beginner

[@maybenew7293]

Excellent course but all due respect, this is for Linux right? I'm still at the minute 5 of the video but it took me 3 hours to sort out how to translate everything in Windows...

[@tylersnard]

You are a clear communicator. Thank you.

[@LoganNinefingers]

Another great video thanks! Is anyone else running into issues when trying this on Windows? Im unable to initialize the database (airflow db init), i get a ModuleNotFoundError: No module named 'termios'..and when i get to run the webserver i get a similar error but instead of termios its "pwd"..anyone aware if there is a way around this on windows or will I have to go with Docker? THanks in advance!!

[@jarathivlogseurope7532]

Hai.. even I am facing the same issue ModuleNotFoundError: No module named 'pwd'.. trying hard to solve this problem but no proper documentation in google too :(

[@LoganNinefingers]

@@jarathivlogseurope7532 I ended up going with Docker, had no luck without it. Hope that helps.

[@zbigniewloboda3393]

2:14 You need to rework the presentation because on small screen of android can't see anything.

[@MrBrykin]

Great video! Very helpful. Do you plan to make new videos about data engineering and Airflow?

[@desarrollojava]

So much help here. You have wonderful skills for teaching.

[@karolinasowinska]

Glad to hear that! ;)

[@prod.bythisjustin8449]

Can you talk about the new Macbook pro with the m1 chip and what it means for developers and machine learning engineers?

[@aureliusnt]

Amazing work, Karolina. You teach very well. Thank you so much.

[@candyfloss184]

I am enamoured by you....you know in s4 hana SAP, we have a FI universal central table called ACDOCA

[@karolinasowinska]

Oh, and what's the purpose of ACDOCA? :)

[@candyfloss184]

@@karolinasowinska instead of many different tables like vendor invoice line items table, customer invoice line item table, vendor invoice document header data table, customer invoice header data table....all tables are now retired and inplace a central ACDOCA table is created.

[@candyfloss184]

This has a benefit that not much customisation is necessary for fetching data from different tables and then displaying same in different reports. Also now sales data is auto reconciled with financial module data.

[@goldenboy2893]

thank you so much this cool and useful video!! :) I have a question. I had some trouble on studying how to use airflow. I want to my code to be 'Idempotently' from just python fuction(def~ for loop + cur.execute(sql query)) to Begin delete from table ~ insert into values(~) END; in this situation, I don't know how to make short duplicated insert sql query. I have a example data frame(csv), that has 100 lines. I tried to make dataframe first and insert into them all in one. But it didn't work. I want put loop method into my BEGIN - END query. plz let me know~~~ ㅠㅠ

[@camilastenico2299]

Thank you! I loved the videos. You explained core concepts in a clear and simple way, well done :)

[@karolinasowinska]

Glad you like them! :)

[@ReenanOFC]

How to connect to the airflow DB in dbeaver?

[@brosnandegenaar4273]

So helpful! Thank you so much for this mini series, I've learnt alot.

[@karolinasowinska]

My pleasure, I'm glad you're finding it useful:)

[@ernestogomez6199]

Hi!! really like your vids, been learning a lot. During this one I've encountered an issue with the dag_folders location. I've changed it a lot of times, but I get 'dev/null'. I've look at stack overflow, but there is not enough info. Do you have any idea of what I should do? I've tried everything in sudo and still the same

[@ismahenelarbi5403]

Hs anyone had to log in first on loca host? I can't log in and do not have access to the dashboard. What did I miss?

[@harshads885]

I tried this code and the DAG fails.Error: "Task exited with return code Negsignal.SIGABRT".

[@andyl9900]

Is airflow standard tool for data engineers?

[@karolinasowinska]

Often yes!

[@lovedeepika7391]

Hey, my airflow webserver is filling dagbag from /dev/null, dag_folder in airflow.cfg and airflow_home both are assigned correctly. I am not being able to solve this thing and because of that airflow is not picking the dags I am inserting in dags folder. could you please help???

[@ernestogomez6199]

Hey I've got the same issue, did you find a way to fix this?

[@bhs88-g2d]

As beginner, i wanna know what os and apps you use...

[@aditi6357]

Could you please make a video for various project ideas ranging from beginner to advanced level? Would be highly grateful :)

[@karolinasowinska]

Great suggestion! That's in my plans :)

[@aditi6357]

@@karolinasowinska Thanks Karolina! Looking forward to it. 🙌