Data Engineering Course for Beginners - #1 EXTRACT

***
My FREE guide on how to become a data engineer:
karolinasowinska.notion.site/...
***
This is the first part of the Free Data Engineering Course for Beginners that I've decided to create for you! Over the course of the four videos, we are going to cover the entire ETL process (extract, transform, load), and at the end we are also going to talk about job scheduling.
In this course you will build your first data feed (or data pipeline) using Spotify API. This feed will run daily, and it will download the data about the songs that you listened to during a day, and save that data in a SQLite database on your local machine.
In this video we are going to cover the Extract stage of the ETL process, which means that we will be learning how to download data from a data vendor (Spotify in this case). Along the way I will also explain many basic data engineering concepts such as data formats (csv, json, XML...), API vs FTP methods of downloading data and the difference between compressed and uncompressed data.
Follow this link to generate your Spotify API token:
developer.spotify.com/console...
Find the code with this data engineering project on GitHub:
github.com/karolina-sowinska/...
-----
If you'd like to learn data engineering, I recommend following the 4 simple steps below to land you the first job interview:
1. Learn Python
I recommend following the Python for Everybody specialization course on Coursera, which is one of the most popular courses there:
imp.i384100.net/x9gVO3
2. Learn SQL
SQL is still the lingua franca of data. I recommend going with Learn SQL Basics for Data Science course, because it contains some chapters which are very releavant to data engineering in partcular, e.g. distributed computing with Spark
imp.i384100.net/QOMZ09
3. Learn Bash scripting/Linux
I wouldn't take a full course on it, but at least read a good article.
if you do prefer to take a course/guided project, I think this one is short and good:
www.coursera.org/projects/com...
4. Learn how to develop on the cloud, e.g. on AWS
There are a few good courses around there, but I think the Coursera one is the most comprehensive
imp.i384100.net/P0MJBM
Discloure: The above contain affiliate links, meaning when you click the links and make a purchase, I receive a commission!
-----
Music:
What Now - Golden Age Radio
Connect with me on Instagram:
@karo_sowinska
And if you'd like to buy me a cup of coffee as a 'thanks'... :)
ko-fi.com/karolina_sowinska

[@lonestarstatechris]

You and Alex the Analyst are my two favorite people to watch. I wish the senior ppl on my team were as clear and concise as yourself. I finished a data science boot camp and got my first job as an analyst, but working remotely. Of course, most of what I do at work wasn’t covered in my boot camp. Your videos have helped a lot!

[@Alexandurs]

This is awesome, thank you so much, Karolina!

[@mezzofemme]

This is great!!! Already looking forward to watching the rest of this series. Thank you!!!

[@karolinasowinska]

Thanks Juan, I'm so glad that you liked it! Next one coming next Thursday, stay tuned! :)

[@karolinasowinska]

My free course is just an introduction to the data engineering world. If you'd like to learn data engineering and be ready to apply for jobs, I recommend following the 4 simple steps: 1. Learn Python I recommend following the Python for Everybody specialization course on Coursera, which is one of the most popular courses there: imp.i384100.net/x9gVO3 2. Learn SQL SQL is still the lingua franca of data. I recommend going with Learn SQL Basics for Data Science course, because it contains some chapters which are very releavant to data engineering in partcular, e.g. distributed computing with Spark imp.i384100.net/QOMZ09 3. Learn Bash scripting/Linux I wouldn't take a full course on it, but at least read a good article. if you do prefer to take a course/guided project, I think this one is short and good: www.coursera.org/projects/command-line-linux 4. Learn how to develop on the cloud, e.g. on AWS There are a few good courses around there, but I think the Coursera one is the most comprehensive imp.i384100.net/P0MJBM Discloure: The above contain affiliate links, meaning when you click the links and make a purchase, I receive a commission!

[@alexisgalarza5969]

Greetings from San Martín de los Andes, Argentina, I love the jazz music in the background in your videos. Thanks for share knowledge!!

[@datamike7457]

Great combination of clear information, comedy, and entertainment. Excellent job!

[@karolinasowinska]

Thanks! I'm glad you liked it ;)

[@thiagoduarte7207]

Just found it! Thanks so much! I'm realy enjoying the series and it's a very useful learning tool for aspiring data engineers like me.

[@AlexAcostaB]

So cool, well done! keep it up!

[@potatoostudies2881]

Thank you for this. You're brilliant!

[@praveenchandkakarla6679]

Thank you so much for this tutorial, especially it was interesting because you have taken real world example which was fun to work with rather than a theoretical serious approach. Kudos!!!!

[@karolinasowinska]

I'm glad that you found it useful and fun! Stay tuned for the next parts! :)

[@maalobie5752]

Great stuff Karolina. I have been looking for something like this for a while. I have subscribed and will be your student for the rest of the series. Thank you

[@karolinasowinska]

My pleasure! :)

[@xtopy9145]

Absolutely the best!

[@thejas795]

This is very instructive and helpfulness. Thanks for share!

[@xx482]

Believe it or not you made my Day... Thanks a lot

[@harshavardhan4523]

The way you explain .....hmmm no words to say 👌👌👌👌

[@iamchesco]

Great video! Looking forward to the next videos of the series.

[@karolinasowinska]

Thanks! Coming next week, stay tuned! :)

[@shovanchowdhury]

Wow ❤️ Your content keeps on improving. Thank you 🙏

[@karolinasowinska]

Ah that means a lot, thanks! If you have any suggestions on how I can keep on improving, let me know! :)

[@iho2437]

Thank you so much Karolina!

[@susieliu8449]

Thanks for sharing Karolina! The instruction is quite clear and helpful!

[@karolinasowinska]

I'm glad to hear that I managed to make them clear, thanks! ;)

[@carlosandrescalvogarcia4866]

You are the beatiful face of data science, and for beginners like me, you have the ability to make it see so easy. Thanks a lot!

[@karolinasowinska]

That's so nice, you're very welcome! :)

[@KP-nc9gk]

You are now my currently favorite human being on KZbin! Thank you!

[@karolinasowinska]

This really is an honour to hear that! I'm glad you enjoy my videos :)

[@RubertoLuca]

Very well done tutorial. I'm pretty new on these topics. Thanks and best regards from Italy

[@paulntalo1425]

Thank you for the video and free course

[@codingtrack4237]

Karolina Sowinska Thank you so much for such a nice explanation (Y) Data Engineering will never be difficult if we have beautiful teachers like you

[@karolinasowinska]

Glad it was helpful! :)

[@cagan8]

Exactly what I needed for my school project!

[@karolinasowinska]

Excellent! I'm glad this is useful! :)

[@alejandragutierrez1709]

This is brilliant, please never stop!

[@danzilbenjamin8196]

agree

[@zeusjean]

agree x2

[@3alaamtwitch753]

agree x3

[@growthmagic8495]

Very easy to understand. Thank you for creating this free course instant subscribe!

[@karolinasowinska]

My pleasure, thanks! :)

[@mwansa430]

Love your videos 😊👍

[@rubenalejandro7763]

This is quality content! Thank you!

[@karolinasowinska]

I'm glad you think so, thank you! :)

[@keyo3945]

what a surprise! yesterday I was thinking to work on my Data Engineering skills and now I can see the course has been started from the loveliest KZbin channel in my subscription list it will also be interesting if you share reviews and book recommendations about the field as always, you nailed it! thanks for the video, I will follow all the course

[@karolinasowinska]

Haha I can read my subscribers' mind! :) I hope that you'll find this course useful! And I'll definitely think about resource recommendations, thanks for the suggestion!

[@mauriciovaler3186]

É disso que o Brasil gosta! So easy to understand! Thank you Karolina 'n thanks youtube algorithm for bringing me here

[@karolinasowinska]

Aw thanks! And thank you KZbin algorithm! ;)

[@BrunoTorres05]

É como dizem, tem BR em todo canto! kkkkkk.

[@sanchessantos7153]

Great video, you explain very well, I learn Data Engineering and English with you didatic! hello from Brazil :)

[@badgyaltiaa]

So Cool, Thank YOU!

[@javiermadriz7834]

I'm Backend Developer but really I like it so much Data Engineering and I think that you teach very well

[@shreedharshekhar1397]

Thanks....this was extremely useful 👍

[@putririzqiyah6294]

i just found your channel and my fingers automatically click on subscribe button without hesitate, nice explanation and easy to understand

[@karolinasowinska]

I'm glad you liked the video! :)

[@mfakhriabdillah]

Thank you for this amazing tutorial!

[@karolinasowinska]

Glad it was helpful! :)

[@ngee4925]

Very cool!

[@JordannGeorge]

Good content! I just started looking into data engineering, which is how I found your channel a moment ago. I like working with the Spotify API so this was right up my alley. Bigger code text would be helpful.

[@karolinasowinska]

Awesome, I hope you'll enjoy the course! And thanks for the feedback, I'll try to make the code larger in the videos that I haven't produced yet! :)

[@ReenanOFC]

Very good video, I'm waiting for the next ones. :D

[@karolinasowinska]

Thanks so much, I'm glad you liked it! The new video's just come out! :)

[@BillyT83]

Clear and concise explanation from our next-door data engineering girl! Looking forward to watching your next video👍

[@karolinasowinska]

I'm glad you liked it! :) Next part coming next Thursday!

[@diahidvegi8536]

I can already see this channel grow exponentially in the next few months.... wanna bet? ;) Aug 20, 2020 - 831 subscribers Dec 28, 2020 - 4.28K subscribers (Updated, but kinda missed the middle months :D)

[@karolinasowinska]

Oh wow, you've come back to this comment to update it, thanks haha! And Happy New Year! :)

[@davidfuentes7806]

I saw that you don't create more videos, please continuee... this videos are amazing

[@lestatJANM]

I loved your sidekick! hahaha

[@vsevapopov2]

Thank you!!!!!! This is what I am looking for on other courses and resources because I don't know what to start with!!!!!!

[@karolinasowinska]

I'm super happy that I can help, I definitely created this course with an aim to help people begin! :)

[@poojankulshreshtha1358]

Thank You So Much Ma'am!!!!!

[@crackintosh]

Your videos are excellent! Make more data engineering videos!

[@Mel-be6mg]

Thank you so, so much for this!! I recently had a Machine Learning internship and during that, I was told that a lot of the things I loved about my internship are very geared towards the data engineering field and I should check it out. I was super excited and eager to work on some projects, but there is nothing for beginners! I love your channel. So much guidance. Thank you :-)

[@karolinasowinska]

I'm very glad that you're enjoying the channel! I hope that this data engineering course will be useful for you then! :) I've just released the second part, if you've finished looking at part one :)

[@Mel-be6mg]

@@karolinasowinska You are seriously amazing. I was at the edge of my seat wondering "Is she going to use Pandas?" I will definitely be watching the entire series. You have helped me confirm that this is the stuff I really loved doing, and want to learn more about. Thank you for all that you do.

[@karolinasowinska]

@@Mel-be6mg I've just noticed your comment. What makes me super motivated to keep releasing new videos are comments like this one :) Thank you so much!

[@Mel-be6mg]

@@karolinasowinska I am seriously honored! Thank you for all of your videos and time. Your advice has helped me find balance and learn so much. :) I am grateful, and I am sure a great many are too.

[@danhle7999]

thank you for making this for free, I appreciate that. Just one thing, you should zoom in to the code so people with a small screen can easily see what you were typing. I cant see a thing

[@karolinasowinska]

Noted!

[@duynghiavo7915]

Anh có thể ib riêng hướng dẫn em cách lấy token API với ạ

[@kevinayers7144]

Ahhhhh this is amazing.

[@karolinasowinska]

I'm glad you think so! :)

[@shibani.palo9]

Trust me when I say this. You are my saviour. I have been wanting to switch jobs and was not really sure how to start preparing. And BAM! I FOUND YOU 😍

[@karolinasowinska]

I'm super happy to read this! If you need some help or have some video suggestions, please give me a shout! :)

[@sprintwithcarlos]

This video is so good it's crazy it has not yet reached 100k visualizations

[@karolinasowinska]

Thank you! Maybe it will one day, who knows haha! :)

[@DroisKargva]

THANKS LOVE YOU

[@nelsongee2025]

Really good video. High recommend.

[@karolinasowinska]

I'm glad it was helpful! :)

[@karissahailee5510]

Love your content

[@karolinasowinska]

I'm so glad that you do! :)

[@Robi2009]

Komentarz dla zasięgu, powodzenia ! ;)

[@karolinasowinska]

A dziękuję! ;)

[@finaafifana4896]

Greatt tutorial. Thank youu 💕 Btw please use another data source such as twitter API for ETL process

[@danielrosas2240]

Hi karolina! :) Beforehand, Your videos are so well explained and your english is so very understandable. On the other hand, I think it would be a great idea to make a playlist of your courses in order to get involved in the course on one click. Just a recommendation! Anyway! Have a great day and keep doing content like these. Greetings! :)

[@karolinasowinska]

Hey! I'm glad you like the videos ;) I actually made a Data Engineering playlist! :) But it also involves a few other data engineering videos, not just the course ones

[@chriskeo392]

Subscribed!!!

[@james4727]

hugggge AM fan here, so seeing that data frame made a very happy man.

[@karolinasowinska]

Oh hey a fellow AM fan ;)

[@nikhilgaur4964]

I am a data scientist and I know I am never going to transition to Data Engineering, but I appreciate what they do, and you do an amazing job explaining the duties of a Data Engineer.

[@nds142]

You are really really doing an amazing job... Keep it worth for a 1M + subs...😀

[@karolinasowinska]

Oh, thank you so much for your support. I hope the course is useful to you! ;)

[@abhishekchakravorty6250]

It was cool Waiting for the next one👍👍👍

[@karolinasowinska]

I'm glad you liked it! The next one coming soon, stay tuned! )

[@TechTraverse519]

Precise and quite informative video. Please bring more technical content related to data engineering. Thanks a lot Carolina ! 😊

[@juanderosas791]

Thanks for th econtent! How can i do if i want to do the same but with podcasts?

[@rodzaniskandar2031]

good artist choice bro

[@jorgemolina6538]

Great video! Just a question, why does she put "sqlite:///" ? Does it points to a specific place or directory? Thanks in advance 🤙🏽🤙🏽

[@hkollaknow4404]

Really you are from the best influencer in KZbin ❤️

[@karolinasowinska]

Aww how very kind of you!

[@yassinefaiq8134]

Thaank you so much !!

[@karolinasowinska]

My pleasure! ;)

[@AahadCyborg]

Awesome tutorial! Also, I see that you have good taste in music ahaha. Artic Monkeys are the best

[@gikis1]

This is great, thank you! Does anyone know how to change scope for the API request to get your recent tracks played?

[@federicococimano8004]

Hi, honestly no , but I want to ask you, how you obtain the token? Because the page change since the release of the video and there are no button to "get token"

[@DGTutos2023]

Great work Karolina 👍👍👍👍

[@karolinasowinska]

Thank you! Cheers!

[@DGTutos2023]

@@karolinasowinska 👍👍❤❤🔥🔥

[@TitovAndrei]

very optimistical gerl

[@gregoriofaerman]

a gold mine

[@potatoostudies2881]

Hello! I tried running your code but I encountered an error regarding the import of python's library-pandas. It was already installed on my computer, however, it gives me an error of Import "pandas" could not be resolved from source. What would you recommend? As a beginner, pardon for the confusion.

[@nicolasrosso3800]

I am a data analyst transitioning (trying to) to data engineering. Thank you for this tutorial. I'm still finding my way in python, so here's my doubt about the code you left: (Line 71) for song in data["items"]: I'm not sure what is ["items"] doing there, can you (or anyone in the comments) help me clarify this? Once again, thank you very much.

[@BillyDTourist]

Thanks for the video Karo. Will make the same API just for practise I understand that this is a more advanced video than the basic one so some assumptions are made e.g. (we can read the json file) It is impotant to highlight the tools you use and the things required in order to do the extraction part e.g. (Python, VSCode, reading json files) Also to someone who is unaware , the numbers you mention at the limit for the spotify API is not something you visibly show how you get. Also maybe highlight the fields you talk about when you mention them in your speech ? Cheers for another nice video. PS I think i don't like the shadows on the "end of paragraph" screens My notification from youtube was late this week =_="

[@karolinasowinska]

Thanks a lot for the detailed feedback, I really appreciate it! I'll try to take in on board in the production of the next videos! Thanks! :)

[@shanilmohan80]

Excellent Tutorial. I don't know who is that one person who disliked this video. I am assuming he works for Amazon music.

[@karolinasowinska]

Thanks a lot! Maybe it was someone from Spotify, since I cancelled their subscription ;))

[@alexandrunknown1456]

and is there a way to understand structure of huge json object. because its not obvious for me how did you understand the exact column names like SONGTITLE etc

[@ericdanielmarruedo1590]

How can I take data from a specific genre? I need the 1000 most popular song of pop in example

[@isaacmolina9818]

For some reason, I'm getting an error. Please help :( in for song in data["items"]: KeyError: 'items'

[@ardiantofahri]

Keep it up, someone's career will be made by your videos :D

[@karolinasowinska]

Aww that's so motivational! Thanks!

[@ardiantofahri]

@@karolinasowinska challenge completed! no need to request token anymore haha

[@ashishvarma9653]

Hi the tutorial is excellent, may I know how to change the datetime to 365 days

[@eanjohnson3537]

I may be a little late to the game, but I love the content. If I could make one recommendation it would be to increase the size of the text in VSCode. Can be kind of hard to read for me at least. Thank you!

[@karolinasowinska]

Never too late! Thanks, I'll keep that in mind for future videos! ;)

[@jxchzp9029]

Thank you so much for this! I keep getting "nan" under artist_names. Same thing happened when I used a different Spotify account. I looked it up and "nan" means "not a number" in numpy but I don't really know what that implies? I used df.dtype and it shows that it is an "object" type, and I tried to convert it to string, nothing happened too. Am I missing anything?

[@santiagobrikman2204]

Thanks a lot for the video! I have a question, why I can't retrieve more than 20 songs? I have chosen 50 as the limit. Any help will be welcomed, thanks in advance!

[@ivanangelino7449]

Hi Karolina, I need contact you for a few tips to conect a redshif database and create a dataframe. do you have any time for these?? Thanks

[@irisothman1041]

Hi! Thank you for making great content especially for people like me who are a newbie in data engineer. I have a question, I got an error at line 1. It wrote there was a ModuleNotFoundError: No module named 'sqlalchemy'. Do you know how to fix it? Thanks for your answer in advance :)

[@karolinasowinska]

Hey! You have to install it in your command line. Run pip install SQLAlchemy

[@jerbear97]

1:47 "you'll be able to look back at all the data you generated" don't forget all the AWS charges you also accumulated

[@nicolasantoniomugasaez9384]

tk's!!!

[@cellculturebabe]

Can you do this tutorial if you haven't already been using Spotify and therefore don't have a Spotify history?

[@finaafifana4896]

hi thank you for the free course. it's very helpful for me. btw can i use jupyter notebook to download the data api?

[@karolinasowinska]

Yes, absolutely:)

[@ryanx2800]

5:05 I curl this link in linux data is null, what happend?

[@user-vo4sq2jx7j]

hey :) Nice edit, got one question about the token part.. I did the same as you and I recieve an eroor saying there is no token provided. what shuld I do?

[@user-vo4sq2jx7j]

error*

[@danzilbenjamin8196]

Nice

[@sanketgupta98]

URGENT: {'error': {'status': 400, 'message': 'Only valid bearer authentication supported'}} I'm getting this on printing data, help me with this error Please

[@haoweilee5813]

Hi Karolina, Thanks for the video. I really like it. One question that need your advice. I tried to view data for more than 20 rows, so I use 'pd.set_option("display.max_rows", 100)' and expected to see 100 rows. But the output still shows me 20 rows only. Is there any way to increase the rows of DataFrame we can explore in this program during the extract data step? Thank you:)

[@karolinasowinska]

It might be that there isn't any more data for that period! Have you listened to more songs than that?

[@chamelious]

This seemed like exactly the kind of video I was after, but then in the python section you just write a bunch of code with little to no explanation. Anyone know where I can go to actually learn this stuff?

[@antonfrombutovo4315]

Karolina, great video) I apologize for belated comment, but while watching the coding part, I noticed something, that caused a question (which is Python related) ) Will be nice if you comment anyhow) On lines 6-7 you first "from datetime import datetime", and then "import datetime". Why do you do both and exactly in that order?)

[@mwansa430]

I have 11 albums by Imagine dragons and I've been listening to them from last year

[@duynghiavo7915]

Can you special instruction for me get token API. I can't how to get token 😢