Thanks for watching! Let me know if you've got any questions! 00:00 - Intro 01:41 - Pre-Requisites 04:59 - Extract and Transform with Python 14:20 - Loading with Lambda & AWS 18:38 - Terraform Deployment 29:13 - Summary
@manuelaguilar96792 жыл бұрын
8⁸
@redrum44862 жыл бұрын
I just want to say thank you my man! I followed this tutorial which took me 2 full dasy to do and it helped me get a jobs as I was able to talk about the process! God bless you homie
@datadrivendecisions02 жыл бұрын
This is why I started making KZbin videos. Love to hear it 🔥congratulations on the job and for putting in the work 💯
@teaspells99943 жыл бұрын
thank you! I loved this video, Im triyng to get a job in the database dev field and ETL is one of the main requirements for these kinds of jobs.
@datadrivendecisions03 жыл бұрын
I’m really glad that this helped you and I hope it helps with the job interviews! I’m definitely going to be doing more videos in the future around the skills required to get into the data/data engineering profession
@sameershaik9998 Жыл бұрын
Successfully done the project. This was my first ETL Project. Thanks a lot liam.😇
@k00a73 жыл бұрын
Thank you for taking time to post this video. I love how you used TF , python and aws tools to build the application.
@datadrivendecisions03 жыл бұрын
You’re welcome Karthik. I’ll be using terraform, AWS and python to deploy my bitcoin bot in my next video!
@k00a73 жыл бұрын
@@datadrivendecisions0 that would be great. Also it would be a great addition if you can use a CI/CD pipline like gitlab.
@datadrivendecisions03 жыл бұрын
@@k00a7 I deploy some of my work with CodePipeline from AWS at the moment. But I also use Jenkins, CircleCi and GitHub actions at work so I can go over any of those in future content if you're interested!
@garcialn Жыл бұрын
Great content, mate. Hope to see more from data engineer Matthew Mcconaughey 😉
@justincooke58882 жыл бұрын
Dude !!! Please Please Please add more content . You make Data Engineering Fun haha !!!
@datadrivendecisions02 жыл бұрын
thank you! check out new content on my new channel kzbin.info
@karolinasowinska3 жыл бұрын
Thanks for so much valuable content in one video! I might finally get my hands on Terraform! P.S. you've just given me AWS instances OCD with that infinite loop story 😅
@datadrivendecisions03 жыл бұрын
Thank you Karolina! Happy too show you some of the basics sometime. I think you know the person who made the infinite loop, we both worked with him 😉
@aaronroethe10102 жыл бұрын
This video was high tier. Excellent Job!
@datadrivendecisions02 жыл бұрын
Thanks Aaron!
@onurbasar78403 жыл бұрын
Thank you. I guess I found a valuable activity to do this evening.
@datadrivendecisions03 жыл бұрын
You’re welcome! I hope you enjoyed your evening!
@shariqshahab53523 жыл бұрын
Great video ! Very informative, yet entertaining content. Looking forward to more cloud computing videos !
@datadrivendecisions03 жыл бұрын
Thank you Shariq! I’ll be deploying my crypto currency trading bot onto AWS soon!
@carlosleotaud32552 жыл бұрын
Damn man you have great attitude and the content is really good, thanks !
@datadrivendecisions02 жыл бұрын
Thanks Carlos! Hope you found it useful
@redrum44862 жыл бұрын
You sir are the Goat.... big up man like Liam!
@mikelmenaba Жыл бұрын
you are a real G
@whateverimnot11373 жыл бұрын
Nice project. I just started to learn terraform, so it was nice to see an example. Would love to see a video about streaming data in AWS and applying some live analytics in the future.
@datadrivendecisions03 жыл бұрын
Glad you liked it! I’ll try and do some more terraform in the future. Could try and come up with a streaming solution in the future. Like using Kinesis for streaming?
@whateverimnot11373 жыл бұрын
@@datadrivendecisions0 Yeah kinesis would be perfect!
@sase10173 жыл бұрын
Good stuff, bro, please make more
@datadrivendecisions03 жыл бұрын
They’re in the pipeline 😏
@raulgutierrez58623 жыл бұрын
Excellent demo!
@datadrivendecisions03 жыл бұрын
Thank you Raul!
@DivineAura1 Жыл бұрын
Thanks for the video! Are you using pycharm btw for the code editor ?
@abhi0603-c3b2 жыл бұрын
Thanks for sharing this excellent video. It shows that you are highly skilled and have in depth knowledge. I have a question as well for you: Would saving the data in RDS have been better for querying with SQL later on? Are there any architectural reasons for choosing S3?
@datadrivendecisions02 жыл бұрын
Yeah an RDS instance would definitely be the better way to store data for analysis. S3 is much quicker and cheaper to setup as it doesn’t require any DDL which is why I chose it for this project
@cjoshmartin3 жыл бұрын
Great video
@datadrivendecisions03 жыл бұрын
Thank you Josh!
@vert_sr2 жыл бұрын
bro you got me dying throughout this whole vid 🤣🤣
@taoufikizem5129 Жыл бұрын
Thank you for this demo, just I didn't get the idea where I should insert the following credentials(Spofify_client_id, spotify_client_secret, S3_name, S3_key..) into API ?
@niklausmikealson3115 Жыл бұрын
I can't understand how the file structure was made. Since it was already created, i am confused what comes first and what not. What to do?
@brandonltran2 жыл бұрын
Fun project! Loved the article too. What IDE are you using in this video?
@datadrivendecisions02 жыл бұрын
Thank you Brandon! Glad you enjoyed it. I was using either PyCharm or IntelliJ. I'd recommend PyCharm for Pyhon :)
@tuetran92123 жыл бұрын
Great channel. I like your enthusiasm. You got a sub from me.
@datadrivendecisions03 жыл бұрын
Thank you Tue! Stay tuned 🤟
@fayssalelaazouziai15732 жыл бұрын
Good stuff, bro , please i have a question , what is the "redirect uri" should i put for this project?
@janakasandun23462 жыл бұрын
Really helpful!..
@datadrivendecisions02 жыл бұрын
Glad it was helpful!
@brianboyle13632 жыл бұрын
Hey Liam hope all is good! I'm doing a college project, basically using your code from the video 'Python ETL Data Pipeline Tutorial with AWS, Terraform and Spotify' I'm just stuck on the artist python code so for an example you have spotify:artist:5K4W6rqBFWDnAN6FQUkS6x but what I'm wondering is where you are getting the 5K... part as I'm doing it with my own playlist.
@datadrivendecisions02 жыл бұрын
Hey Brian, we chatted on Instagram and you found your answer by right clicking on the albums to get the artist links ☺️
@maybenew7293 Жыл бұрын
damn... it took me 3 hours just only for installing all the packages... 😮💨
@dbseraph3 жыл бұрын
The answer is contracts and attention spans.
@datadrivendecisions03 жыл бұрын
😂
@datadrivendecisions03 жыл бұрын
What’s new in the contracts these days?
@arunkumaracharya96412 жыл бұрын
This can be a 1-month crash course for a beginner even if he knows python and AWS.
Жыл бұрын
Hello, thank you very much for this project! I made a project based on yours, but now I'm stuck. I created the project and used Terraform to deploy to AWS. But when trying to run the file with the collect_data function in AWS Lambda, I got this error: "[ERROR] Runtime.ImportModuleError: No module named 'redis' Traceback" Do you have any idea what could be causing this error? Thank you.
@SameerShaik-ic1oz Жыл бұрын
same problem
@sase10173 жыл бұрын
What IDE are you using? IntelliJ?
@datadrivendecisions03 жыл бұрын
IntelliJ in this video but I would recommend using PyCharm! I use it in my more recent videos
@tello95043 жыл бұрын
Why keep it in a Lake and not in a Warehouse?
@datadrivendecisions03 жыл бұрын
Purely for simplicity and costs. A properly constructed data warehouse would definitely be the better end product!
@tello95043 жыл бұрын
@@datadrivendecisions0 Thank you
@datadrivendecisions03 жыл бұрын
Thank you José!
@firefoxcarranza64322 жыл бұрын
Plis, make tutorial from Sagemaker, thanks..
@ChhatreMihir3 жыл бұрын
Loved the video! Thanks got making this. Would love to see some more videos around building data pipelines :) Sub from me 👍
@datadrivendecisions03 жыл бұрын
Thank you Mihir! I'll be finishing off this crypto series and then I'll be talking about my football betting algorithm which is effectively a pipeline of data with some logic built on top of it! Stay tuned :)
@DaarioNeharis2 жыл бұрын
implemented almost everything, but getting the following error when trying to run terraform apply: AccessDeniedException: User: arn:aws:iam::16003352######:user/spotify-data-analysis is not authorized to perform: events:DescribeRule on resource