Airflow for Beginners - Run Spotify ETL Job in 15 minutes!

In this long-awaited Airflow for Beginners video I'm showing you how to install Airflow from scratch, and how to schedule your first ETL job in Airflow!
We will run the Spotify ETL job that we created in my Data Engineering Course for Beginners. But if you have your own code to schedule in Airflow - feel free to use it instead!
In this Airflow tutorial I just wanted to get you started with Airflow, because I know that configuring Airflow can be non-trivial. I won't be going in-depth into all Airflow's features, so hopefully this tutorial will be as beginners friendly as it can get!
I'll show you how to fix some installation errors, how to set up your project folders, how to edit Airflow config file and where it sits, and how to write your first DAG file!
Get ready for a ride!
Oh, but grab some useful links first:
-github code repo: github.com/karolina-sowinska/...
-Airflow quick start guide: airflow.apache.org/docs/stabl...
-Spotify access token: developer.spotify.com/console...
Don't forget to subscribe if you're new here, more content is coming!
:)
---------
If you'd like to learn data engineering, I recommend following the 4 simple steps below to land you the first job interview:
1. Learn Python
I recommend following the Python for Everybody specialization course on Coursera, which is one of the most popular courses there:
imp.i384100.net/x9gVO3
2. Learn SQL
SQL is still the lingua franca of data. I recommend going with Learn SQL Basics for Data Science course, because it contains some chapters which are very releavant to data engineering in partcular, e.g. distributed computing with Spark
imp.i384100.net/QOMZ09
3. Learn Bash scripting/Linux
I wouldn't take a full course on it, but at least read a good article.
if you do prefer to take a course/guided project, I think this one is short and good:
www.coursera.org/projects/com...
4. Learn how to develop on the cloud, e.g. on AWS
There are a few good courses around there, but I think the Coursera one is the most comprehensive
imp.i384100.net/P0MJBM
Discloure: The above contain affiliate links, meaning when you click the links and make a purchase, I receive a commission!
Find me on Instagram:
@karo_sowinska
You can also warm my heart with a cup of coffee as a thanks!... :)
ko-fi.com/karolina_sowinska

[@ElPapelMan]

One of the best etl series I've ever watched on youtube... thank you.

[@tejagoud4871]

Watching this is really worth the time. Not like other KZbin channels where they run promotions for a minute or two. Above all, it is really a good video on getting started with Airflow. Great work Karolina. You are an amazing instructor.

[@karolinasowinska]

I really appreciate this comment, thanks so much Teja!

[@ricardoarbois2839]

@@karolinasowinska Nice video.i really love your installation., hope you dont mind if i post here my yt vid about installation airflow in heroku....thanks and please more vid in airflow...kzbin.info/www/bejne/jpKXg6pjmMd-f7M

[@zma314125]

Thank you for explaining such a complicated topic in a simple way. This will definitely be a help as a foundation to data engineering. Keep up the great work!

[@ChernobylPizza]

I like your tutorials because they are simple. I kinda got stuck biting off more than I can chew and I was going in circles for a while. Data careers are about so many things it's easy to get lost (python, hadoop/spark, airflow, ML, math/stats, visualization, cloud).... I just needed something simple I can do easily to get going.

[@itsrainingcatsanddogs]

I'll be needing more of these airflow tutorial

[@karolinasowinska]

I'll try to do my best! :)

[@Marc_B.4]

The video I was waiting for! I'm happy to see it, it's very well presented. It was really useful, I now have a good feeling about how Airflow works. Can't wait to see what's next on your channel :)

[@karolinasowinska]

Aw I'm super glad to hear that it met your expectations! Thanks! :)

[@chrish.4734]

Great video, thanks a lot Karolina! I really like your clear way to explain, which is straight to the point and your great energy!

[@geetanshkumar1854]

Amazing work ma'am. I am new to all this and this tutorial was so simple and clear. Your way of explaining is also unique because you talk about errors as well which very few people do.

[@karolinasowinska]

Thank you, I'm glad you like my talking style! :)

[@ashishk81]

i am data scientist by profession and wanted to learn data engineering in details , i didnt found single free online resource to learn all data engineering skills ..you are doing great job ..waitng for your videos

[@karolinasowinska]

I'm super glad my videos are useful! :)

[@ashishk81]

@@karolinasowinska Can you please suggest free online resources to learn end to end data engineer

[@brosnandegenaar4273]

So helpful! Thank you so much for this mini series, I've learnt alot.

[@karolinasowinska]

My pleasure, I'm glad you're finding it useful:)

[@MrDavisv]

Best explanation I’ve see about DAGs and super helpful intro to Airflow. Makes complete sense. Thank you!

[@karolinasowinska]

I'm glad this was helpful! :)

[@yuriershov6530]

Great content! Just what I needed before starting my data engineering courses

[@karolinasowinska]

Glad it was useful! :)

[@Nedwin]

Love the extract, transform, load part! ❤️

[@imdadood5705]

Something that I was looking for. I know Python, SQL, R and a good amount of machine learning. But I didn’t know what to do next. I just search for Apache Airflow and I got this! Thank you!

[@quinnluong114]

She a real one, you can tell because she's showing all the problems she's running into

[@josecarlossilva3670]

Awesome content!! I ve been struggling with that stuff for few months. Thanks for sharing

[@nataindata]

Karolina, thank you a lot for your efforts and for making these videos! You've sparkled ingenuine interest in me to try the project out. Plus, it's really great to know that Data Engineering community is empowered by women. I'm only starting my way in DE, so it's great to follow you and learn. Love ❤️

[@karolinasowinska]

Thanks so much for this lovely comment! Good luck on your DE journey and I hope I'll see you around here! :)

[@samuelabolo]

"back to downgrading our future" got me cracked up

[@datexland]

You're really good to explain the basic concepts, Thanks so much for sharing, definitely you gonna win a lot of subscriptors

[@HenriqueP]

Karolina, I was enlightened by your explanation/methodology, helped me a lot to get started with Apache Airflow, mad props for this! Keep up with the work! Cheers from Brazil

[@karolinasowinska]

I'm so glad that my effort didn't go to waste! Thanks for your comment! :)

[@mehdiyahiacherif2326]

well , i had a BI (business intelligence) project this year and i had no idea what are those etl and reporting tools , i searched for like 10 days and tested a lot of softwares some of them were usefull and some of them were ... meh and i actually liked airflow and dbeaver and both are in this video what a surprise , for people who wants to test some bi tools tou have (free and open source) : etl :airflow , knime , pentaho DI reporting : superset,also some cool dashboards in pentaho server db RAT and gui tools : dbeaver and also DbSchema data mining : tanagra and weka take a look also at apache kylin (i did not knew how to setup it to get postgres as a datasource so ...) good luck guys and great video lady

[@karolinasowinska]

Awesome, thanks so much for the tips! :)

[@mehdiyahiacherif2326]

@@karolinasowinska thanks for your reply , I would be very happy if you make a video of how to connect apache kylin to nex data sources ( it is painfull lol I searchedand itis not well documented) A serie about bi tools and data manipulation can be a great idea since not a lot of ppl do it on youtube Good luck

[@Paperwood360]

What a great tutorial, the best I've seen so far for Airflow! Thank you very much

[@rodrigoamoedo8523]

your content is getting better every time

[@karolinasowinska]

Oh, really? I'm super glad you think so! :)

[@BestevertechBlogspot]

Thank you for posting such relevant content. These are really worth it!

[@karolinasowinska]

My pleasure! :)

[@AlexAcostaB]

This is such a great introduction to Airflow. I already designed one pipeline and I am ready to implement it. Thank you so much.

[@karolinasowinska]

That's fantastic, how did it go? :)

[@AlexAcostaB]

@@karolinasowinska it work well. I’m getting data from one customer’s FTP and it is failing using Python 🐍 working on a solution and it will be ready for deployment

[@karolinasowinska]

@@AlexAcostaB Awesome stuff :)

[@aureliusnt]

Amazing work, Karolina. You teach very well. Thank you so much.

[@Tien-Tjie]

oh my god I was waiting for this

[@karolinasowinska]

Aw, I'm super glad to hear that! ;) I hope you enjoyed it!

[@camilastenico2299]

Thank you! I loved the videos. You explained core concepts in a clear and simple way, well done :)

[@karolinasowinska]

Glad you like them! :)

[@McCallumClips]

Your video was exactly what you said it would be. An introduction. VERY GOOD JOB! Thank you.

[@karolinasowinska]

Yes indeed, it'd be very hard to discuss details in a 15-minute video! I'm glad you liked it! :)

[@sirosala]

Excellent Karo !!!! 💪💪💪

[@AthanasiouApostolos]

Love your tutorials. Well done :)

[@karolinasowinska]

Thanks so much! :)

[@srh1034]

Wow, simply amazing!

[@pranoygowda4595]

Could get on the concepts and working with Airflow just by watching the video. Very much helpful video to get started with. Amazing!

[@karolinasowinska]

Glad you enjoyed it!

[@josesebastiancolaneri7125]

Excellent video! Thank you very much Karolina!

[@marvhan888]

what a well done mentor's job you are doing.

[@karolinasowinska]

Thank you!

[@buithanhlam3726]

I found it very hard in the documentation, the book... then I found your video. Thanks a lot!

[@McMurchie]

Great video Karolina, for those struggling with pip install - I suggest doing a quick learn of conda so you can create a quick conda environment to install airflow without messing up your primary python/pip libs and versions. I agree though, Airflow is so tricky to set up.

[@OPopoola]

Thanks for this introduction. I have been wondering what the big deal is with Airflow. Now I see the potential.

[@karolinasowinska]

My pleasure! :)

[@alexanderbenavides1887]

Wuau, an amazing video tutorial. I love your videos :)

[@SASUKEUCHIHA-yc6er]

The moment I saw the video Thought she has over a million subscribers She deserves more subscribers and also more views....

[@karolinasowinska]

Oh, that's very nice to hear, thanks! :) 🙏

[@tylersnard]

You are a clear communicator. Thank you.

[@edragon1412]

This is really helpful for Airflow beginners like I am. I am appreciated your work a lot. Keep working those topic like this, girl ;)

[@karolinasowinska]

I'm glad you enjoyed it! I will keep it up ;)

[@migueldias1292]

very cool video! well done!

[@DDAN48LIFE]

I love Karolina , you are the best

[@heikokraemer2735]

Thank you Karolina, very useful, totally no waste of time.

[@karolinasowinska]

I'm glad it was useful! ;)

[@akshayshenoy6827]

Thank you so much Karolina! I learnt alot today by watching your video. :)

[@romanlukichev4971]

This is the first time I've heard about Airflow. Apache Airflow is an open-source workflow management platform for data engineering pipelines. It started at Airbnb in October 2014[2] as a solution to manage the company's increasingly complex workflows. Creating Airflow allowed Airbnb to programmatically author and schedule their workflows and monitor them via the built-in Airflow user interface.[3][4] From the beginning, the project was made open source, becoming an Apache Incubator project in March 2016 and a top-level Apache Software Foundation project in January 2019. Airflow is written in Python, and workflows are created via Python scripts.

[@dnrocha1]

Great content! Thanks, it was very helpful!

[@Shagysami]

Thank you for the neat video ! I'm new to data engineering and may be nailing my upcoming job interview thanks to you

[@karolinasowinska]

Wow fingers crossed!;)

[@hisky74]

The nano tip is very useful!! Very good content! Thank you!

[@karolinasowinska]

Glad it was helpful!

[@marianosin93]

Great job!!!

[@nskeip]

Nice video. And about direc acyclic graphs - actualy, you could draw an arrow from 3 to 2 in the graph you showed as an example ^_^ (because there was no way to go back from 2 to 3, so it would not make a cycle)

[@karinaahmedova5221]

superb explanation! thanks

[@NewyJimmy]

More on airflow please! This was great!

[@karolinasowinska]

I'll see what I can do! I'm glad you enjoyed it! ;)

[@AndresHernandez-mz3xh]

Wow, thank you so much Karolina! It helped me a lot with my project!

[@karolinasowinska]

My pleasure!

[@IppoCrypto]

love your work

[@karolinasowinska]

Thanks!

[@desarrollojava]

So much help here. You have wonderful skills for teaching.

[@karolinasowinska]

Glad to hear that! ;)

[@ahmedtremo]

I love you, thanks for the video

[@TaylorNelson1]

Ah the joys of finding new errors when you try to install things for a new production environment... this is such an accurate depiction of real engineering life.

[@karolinasowinska]

Exactly! ;)

[@TJ-hs1qm]

I wasn't looking for airflow nor did I know what airflow even is but stayed the whole video.... know I know 🤣

[@baotran4175]

Thanks very much. I come from Vietnam. Right now intern Data Engineer. I hope you can do more topics on Data Engineers in the near future

[@karolinasowinska]

Hello there! Nice to hear from a fellow techie. I will do for sure! ;)

[@gcmmartinsleme]

Thanks. It was very helpful for me!

[@ankushojha5089]

thanks for each & every videos 👍

[@karolinasowinska]

My pleasure!! :)

[@resap.9128]

Great one again, Thank you

[@karolinasowinska]

My pleasure! :)

[@marcosmel1087]

YOU ARE AMAZING!

[@AbdurrahmanKocukcu]

Great video 🤗

[@jae11011]

Great video for airflow beginner!! I have tried to run airflow for too much time and always stuck even before start webserver. This is the first time I successfully run it!! For anyone who is also new with Airflow, I got some small issue when I follow with the video. Here is how I solve it, just in case anyone encounter the same issue. To start airflow 1. After install airflow, need to run airflow first, to create the airflow.cfg in the home.(If you haven't run it before) Simply type "airflow" will do the work. I didn't run it first, so I couldn't find the cfg file anywhere. 2. I also need to run "airflow db init" to create the db for logs. 3. Last, I need to create a user before I use the webserver, otherwise there will no user for me to login. These steps are available in the airflow document quick start as well. To run the dag as in the video. 4. I switched the toggle to ON in the dag view, otherwise the task will remain in running forever. 5. To run the extract.py or run_spotify.py, I need to put extract.py in the dags folder first. I just put the file directly in the dag folder, but I saw others put the whole python package(subfolder with __init.py) in the dag folder. The latter approach is better for bigger project with no doubt. But I still want to know does everyone put the packages directly in dag folder in real world? Since it's still a little messy for me putting dags file with scripts itself. A few questions I have though, should I terminate and restart airflow scheduler everytime I change my script or it will pick up automatically? I am still having token expired issue when run the script in airflow, even I updated my token in the script and ran fine in local machine. But it is a awesome video to me! Thanks to Karolina!

[@manoharannadar9499]

Best.... keep posting 🙏🙏

[@jparrax1]

AMAZING VIDEOOOOOOO! Thank you for your time!

[@karolinasowinska]

Thank you!! :)

[@prafulmaka7710]

Very nice!!

[@sanchesrfl]

DAG = a directed collection of tasks without going back. Thanks!!!!!!!!!

[@karolinasowinska]

You can remember it this way too :)!

[@xtopy9145]

You are the best, Thank you!

[@Paulo-oy1gs]

great stuff! thanks

[@avinandanbanerjee9568]

Important note - You won't find the airflow directory until you run something on the CLI using airflow Just type in airflow once and hit enter to find the config file

[@kelvin5685]

Thank you! Also one needs to run "source airflow-venv/bin/activate" before running the command "airflow". That way you don't get an error that "airflow" command is not found

[@andizvlogs]

I love your channel, tahnk you so much, I am learning a lot.

[@karolinasowinska]

I'm super glad to hear that! :))

[@kadourkadouri3505]

Very concise nice tutorial ! Thanks

[@foxohnfire]

Thanks Karol :)

[@blakegirardot5813]

Thank you very much, this style of quick start works really well for me, thank you for making it and sharing it!

[@karolinasowinska]

Glad it helped!

[@blakegirardot5813]

Also, before the scheduler would run, after installation, I had to run `airflow db init` the the scheduler started right up.

[@xmagcx1]

I Love it!

[@avinashbasetty]

Thank you for the Spotify api demo. Luckily I have installed airflow 2.0 version today in my ubuntu 18.04 with no errors.

[@karolinasowinska]

Oh wow, looks like I was unlucky with my environment setup, good to hear that your installation went smoothly!

[@troymann5115]

Nice video! One thing to think about concerning running Docker containers from the DAG: Airflow 1.x apparently has an issue which leaves containers in a non-started state. (At least it was a problem in our environment.) Airflow 2.0 seems to have resolved it. Thank you for making this video.

[@arjunjadhav3062]

Love you and thanks.

[@remi954]

Thank you very much!

[@Grace-ql7pv]

I’d love more Airflow videos

[@karolinasowinska]

I'll see what I can do :)

[@Indianvloggerinfinland]

Hi Karolina , Thank you for the video. I need a help as am kinda stuck at one place where you will be editing the airflow.cfg file. I use Macbook. I couldnt find the file even after installing the aiflow. I dont see the folder airflow at all in spite of giving the command "export AIRFLOW_HOME=/airflow" . Need your help on this.

[@PenStab]

Really impressive explanations and teaching approach. You were concise but covered so many small in-between points that I would have otherwise missed. I'm definitely subscribing and going to watch other videos! My only complaint would be the resolution of the capture of the VS Code window - can it be a 16:9 ratio? It was so small on my phone.

[@karolinasowinska]

Thanks! I'll try to improve resolution going forward ;)

[@LaDivisionDeAriza]

Greetings from Colombia. Hey, great video. :) It would be cool if you talk about the tools you use the most, but rather than as tutorials as explanations of what does each one, in which cases is useful, what are the things you like and the thinks you don't about them... like a basic explanation of your preferred tool set, this would be very useful.

[@karolinasowinska]

Hey Emmanuel, that's a great idea! I was thinking about it myself too. I think I'm going to do some videos along these lines ;) Thanks!

[@rawcodes]

Thank you 💕

[@user-cw7wt1fc5j]

More more more courses on data engineering pleaseeeeee. 🥰🥰🥰🥰🥰

[@karolinasowinska]

In progress! :)

[@randolphralph8322]

This is a great tutorial. I am having difficulties setting this up in Windows 10 environment. I was able to setup the virtual environment, but the install process for Airflow differs.

[@veronicalima5664]

You so Funny! I like soooooo much your video! Thanks!!! 😍😍😍😍❤️❤️

[@karolinasowinska]

My pleasure! :)

[@jpb2085]

Fantastically well explained, thanks for sharing. Shame Airflow such a pita to install, easiest route I found via docker option. Subbed

[@karolinasowinska]

Thanks! :) Shame indeed... I guess it depends on your environment, but it was a pain for me for sure haha!

[@davidlr97]

just watched your short data engineering course, very helpful intro to the topic. Thought I should mention that the playlist is out of order, though.

[@karolinasowinska]

Thanks for that! I'll try to fix the order

[@akbarazad93]

Thanks Karolina! I have an issue when replicating this pipeline's framework. It's regards to the ETL part. I was able to run the DAG successfully and when I went into my sqlite database, the table was not updated (no new records appended) from the DAG run. Any idea what could be the issue? I'm using Python 3, Ubuntu on Windows OS. Thanks!

[@solagan7402]

Hello, you agree that the laptop where you configure airflow etc.. needs to be on to run the etl daily ? If the laptop is off nothing will happen right ?

[@ernestogomez6199]

Hi!! really like your vids, been learning a lot. During this one I've encountered an issue with the dag_folders location. I've changed it a lot of times, but I get 'dev/null'. I've look at stack overflow, but there is not enough info. Do you have any idea of what I should do? I've tried everything in sudo and still the same

[@don.geo.thayyil]

for second in video: print("Thank You Sooo Much. Please do more videos

[@karolinasowinska]

More to come! ;))

[@sirgalahad1470]

14:02 Great taste in music 👍

[@fajarabdulkarim7672]

Hi carolina, thanks for nice video. Easy to understand. Btw do you know how to test the dag ? Its Like unit test or the test which QA did in software development

[@GustavoLeig]

Thanks for this awesome tutorial Karolina, one question, the first time I run, the Airflow starts lots of jobs, and the final table gets 20 songs, is that correct? Why so many jobs to get 20 rows?