Python ETL Data Pipeline Tutorial with AWS, Terraform and Spotify

[@datadrivendecisions0]

Thanks for watching! Let me know if you've got any questions! 00:00​ - Intro 01:41 - Pre-Requisites 04:59 - Extract and Transform with Python 14:20 - Loading with Lambda & AWS 18:38 - Terraform Deployment 29:13​ - Summary

[@manuelaguilar9679]

8⁸

[@redrum4486]

I just want to say thank you my man! I followed this tutorial which took me 2 full dasy to do and it helped me get a jobs as I was able to talk about the process! God bless you homie

[@datadrivendecisions0]

This is why I started making KZbin videos. Love to hear it 🔥congratulations on the job and for putting in the work 💯

[@teaspells9994]

thank you! I loved this video, Im triyng to get a job in the database dev field and ETL is one of the main requirements for these kinds of jobs.

[@datadrivendecisions0]

I’m really glad that this helped you and I hope it helps with the job interviews! I’m definitely going to be doing more videos in the future around the skills required to get into the data/data engineering profession

[@sameershaik9998]

Successfully done the project. This was my first ETL Project. Thanks a lot liam.😇

[@k00a7]

Thank you for taking time to post this video. I love how you used TF , python and aws tools to build the application.

[@datadrivendecisions0]

You’re welcome Karthik. I’ll be using terraform, AWS and python to deploy my bitcoin bot in my next video!

[@k00a7]

@@datadrivendecisions0 that would be great. Also it would be a great addition if you can use a CI/CD pipline like gitlab.

[@datadrivendecisions0]

@@k00a7 I deploy some of my work with CodePipeline from AWS at the moment. But I also use Jenkins, CircleCi and GitHub actions at work so I can go over any of those in future content if you're interested!

[@garcialn]

Great content, mate. Hope to see more from data engineer Matthew Mcconaughey 😉

[@justincooke5888]

Dude !!! Please Please Please add more content . You make Data Engineering Fun haha !!!

[@datadrivendecisions0]

thank you! check out new content on my new channel kzbin.info

[@karolinasowinska]

Thanks for so much valuable content in one video! I might finally get my hands on Terraform! P.S. you've just given me AWS instances OCD with that infinite loop story 😅

[@datadrivendecisions0]

Thank you Karolina! Happy too show you some of the basics sometime. I think you know the person who made the infinite loop, we both worked with him 😉

[@aaronroethe1010]

This video was high tier. Excellent Job!

[@datadrivendecisions0]

Thanks Aaron!

[@onurbasar7840]

Thank you. I guess I found a valuable activity to do this evening.

[@datadrivendecisions0]

You’re welcome! I hope you enjoyed your evening!

[@shariqshahab5352]

Great video ! Very informative, yet entertaining content. Looking forward to more cloud computing videos !

[@datadrivendecisions0]

Thank you Shariq! I’ll be deploying my crypto currency trading bot onto AWS soon!

[@carlosleotaud3255]

Damn man you have great attitude and the content is really good, thanks !

[@datadrivendecisions0]

Thanks Carlos! Hope you found it useful

[@redrum4486]

You sir are the Goat.... big up man like Liam!

[@mikelmenaba]

you are a real G

[@whateverimnot1137]

Nice project. I just started to learn terraform, so it was nice to see an example. Would love to see a video about streaming data in AWS and applying some live analytics in the future.

[@datadrivendecisions0]

Glad you liked it! I’ll try and do some more terraform in the future. Could try and come up with a streaming solution in the future. Like using Kinesis for streaming?

[@whateverimnot1137]

@@datadrivendecisions0 Yeah kinesis would be perfect!

[@sase1017]

Good stuff, bro, please make more

[@datadrivendecisions0]

They’re in the pipeline 😏

[@raulgutierrez5862]

Excellent demo!

[@datadrivendecisions0]

Thank you Raul!

[@DivineAura1]

Thanks for the video! Are you using pycharm btw for the code editor ?

[@abhi0603-c3b]

Thanks for sharing this excellent video. It shows that you are highly skilled and have in depth knowledge. I have a question as well for you: Would saving the data in RDS have been better for querying with SQL later on? Are there any architectural reasons for choosing S3?

[@datadrivendecisions0]

Yeah an RDS instance would definitely be the better way to store data for analysis. S3 is much quicker and cheaper to setup as it doesn’t require any DDL which is why I chose it for this project

[@cjoshmartin]

Great video

[@datadrivendecisions0]

Thank you Josh!

[@vert_sr]

bro you got me dying throughout this whole vid 🤣🤣

[@taoufikizem5129]

Thank you for this demo, just I didn't get the idea where I should insert the following credentials(Spofify_client_id, spotify_client_secret, S3_name, S3_key..) into API ?

[@niklausmikealson3115]

I can't understand how the file structure was made. Since it was already created, i am confused what comes first and what not. What to do?

[@brandonltran]

Fun project! Loved the article too. What IDE are you using in this video?

[@datadrivendecisions0]

Thank you Brandon! Glad you enjoyed it. I was using either PyCharm or IntelliJ. I'd recommend PyCharm for Pyhon :)

[@tuetran9212]

Great channel. I like your enthusiasm. You got a sub from me.

[@datadrivendecisions0]

Thank you Tue! Stay tuned 🤟

[@fayssalelaazouziai1573]

Good stuff, bro , please i have a question , what is the "redirect uri" should i put for this project?

[@janakasandun2346]

Really helpful!..

[@datadrivendecisions0]

Glad it was helpful!

[@brianboyle1363]

Hey Liam hope all is good! I'm doing a college project, basically using your code from the video 'Python ETL Data Pipeline Tutorial with AWS, Terraform and Spotify' I'm just stuck on the artist python code so for an example you have spotify:artist:5K4W6rqBFWDnAN6FQUkS6x but what I'm wondering is where you are getting the 5K... part as I'm doing it with my own playlist.

[@datadrivendecisions0]

Hey Brian, we chatted on Instagram and you found your answer by right clicking on the albums to get the artist links ☺️

[@maybenew7293]

damn... it took me 3 hours just only for installing all the packages... 😮‍💨

[@dbseraph]

The answer is contracts and attention spans.

[@datadrivendecisions0]

😂

[@datadrivendecisions0]

What’s new in the contracts these days?

[@arunkumaracharya9641]

This can be a 1-month crash course for a beginner even if he knows python and AWS.

Hello, thank you very much for this project! I made a project based on yours, but now I'm stuck. I created the project and used Terraform to deploy to AWS. But when trying to run the file with the collect_data function in AWS Lambda, I got this error: "[ERROR] Runtime.ImportModuleError: No module named 'redis' Traceback" Do you have any idea what could be causing this error? Thank you.

[@SameerShaik-ic1oz]

same problem

[@sase1017]

What IDE are you using? IntelliJ?

[@datadrivendecisions0]

IntelliJ in this video but I would recommend using PyCharm! I use it in my more recent videos

[@tello9504]

Why keep it in a Lake and not in a Warehouse?

[@datadrivendecisions0]

Purely for simplicity and costs. A properly constructed data warehouse would definitely be the better end product!

[@tello9504]

@@datadrivendecisions0 Thank you

[@datadrivendecisions0]

Thank you José!

[@firefoxcarranza6432]

Plis, make tutorial from Sagemaker, thanks..

[@ChhatreMihir]

Loved the video! Thanks got making this. Would love to see some more videos around building data pipelines :) Sub from me 👍

[@datadrivendecisions0]

Thank you Mihir! I'll be finishing off this crypto series and then I'll be talking about my football betting algorithm which is effectively a pipeline of data with some logic built on top of it! Stay tuned :)

[@DaarioNeharis]

implemented almost everything, but getting the following error when trying to run terraform apply: AccessDeniedException: User: arn:aws:iam::16003352######:user/spotify-data-analysis is not authorized to perform: events:DescribeRule on resource