Superstore Data Analysis | End to End AWS Data Engineering Project for Beginners

In this video we will build a end to end data engineering project using AWS. This project will use following AWS services:
1- IAM
2- S3
3- AWS Glue
4- AWS Athena
5- AWS Quicksight
Here is the Kaggle dataset: www.kaggle.com...
Steps to create AWS account: repost.aws/kno...
Start your Data Analytics Journey : www.namastesql...
Zero to hero(Advance) SQL Aggregation:
• All About SQL Aggregat...
Most Asked Join Based Interview Question:
• Most Asked SQL JOIN ba...
Solving 4 Trick SQL problems:
• Solving 4 Tricky SQL P...
Data Analyst Spotify Case Study:
• Data Analyst Spotify C...
Top 10 SQL interview Questions:
• Top 10 SQL interview Q...
Interview Question based on FULL OUTER JOIN:
• SQL Interview Question...
Playlist to master SQL :
• Complex SQL Questions ...
Rank, Dense_Rank and Row_Number:
• RANK, DENSE_RANK, ROW_...
#aws #dataengineer #project

[@ankitbansal6]

I will be doing more complex projects in the future. Please support this initiative by liking the video and sharing it with your friends.

[@rahularora3832]

Will you create only projects or as a pre-requisite you may dig deep into particular topics related to aws data engineering first and then create projects in which those topics would be used? BTW, thank you for sharing this. I liked it.

[@AkshayaReddyKoduri]

It will be helpful if you do a project on AWS using Glue S3 Lambda and Redshift

[@tejasnaikwade9193]

more complex projects will br very helpful sir.

[@prashantsingh6260]

Thank you, Ankit, for the informative and approachable contents to kickstart into AWS for beginners... There's much more to delve into for real-world challenges, but your dedication to taking time from your busy schedule to assist peoples is truly commendable... Your support for needy peoples is invaluable. Thanks and keep up the great work!

[@ankitbansal6]

More to come 🫴

[@kaushaldubey140]

Good video for starting with AWS glue. I am looking forward for end to end ETL project using AWS glue and pyspark where we will be extracting data from a database (eg. MySQL), storing it into S3, doing some transformation on it and finally loading it into Redshift. Request you to please create one such project video.

[@ankitbansal6]

Great suggestion!

[@sakshibakare9716]

Thank you , Ankit this very useful for AWS beginners. your videos are always helpful for interview

[@ankitbansal6]

Glad to hear that😊

[@vishwassharma3312]

Sir please upload complete play list on AWS and BIG Data. 🙏🙏

[@ketanjain93]

Hello Ankit, I was facing Problem in Amazon Quiksight , when i create charts in it , it is giving error "the data source or data set was deleted or become unavailable while Amazon Quicksight for ingesting the data into Spice. you can import a data to try again". what wrong i have done?

[@anshumanmahapatra9064]

Hi Ankit Thank you for the insight on different AWS tools. I just wanna know how Glue crawler will work in a database with multiple tables where they are related with Primary foreign keys etc, I mean does it crawl the entire schema? Also if possible make videos on how Glue is used in ETL process and can we load to AWS Redshift using Glue?

[@prashantmhatre1328]

Really like this project Ankit Bhai, Thank you very much

[@ankitbansal6]

Thanks for liking😊

[@sriharinair227]

Hey Ankit, any way to automate - 1) Creating different csv files for each date. 2) Moving each file into a separate folder. Any help will be appreciated!

[@gururajhdesai4854]

Thank you. Create more basic projects.

[@dillikaruru254]

It's a great initiative Ankit. You explained the project in a very detailed way and I am able to understand it easily. Crisp and clear. Please do more projects based on Glue jobs, SES, SNS, lambda, cloud formation, CI/CD for a project with github actions etc.

[@ankitbansal6]

Sure

[@Ashu23200]

In need of snowflake end to end project 😊

[@rohitsaswadkar]

Great guidance I ever scene in field of AWS. I completed that project successfully. You gave best guidance while teaching. I also viewed some youtubers but their guidance worst as when you are at mid of the project some error happens and they skip that process and you stuck. Made more videos on Data Engineering. We want your guidance sir.......Love from Maharashtra

[@priyaghosh9971]

I am getting access denied to create the crawler, even if the regions are same and also giving the necessary polices ,if anyone faced the same error please can you tell how to resolve it

[@swapnam1528]

for me partitions are not getting created. What could be the reason.

[@JamiewithData]

Appreciate your effort.

[@abiramiarivazhagan1467]

Can we upload this project on github?

[@shivam87480]

how to showcase this project, how to upload on github

[@naveengvkvs1915]

Good video for starters👍

[@sagarvajsharwa]

Great work 👍 thank you for sharing this

[@ankitbansal6]

Thank you! Cheers!

[@vivek29925]

Great video Ankit. Hope to see more real world project based videos from you. Thanks for your efforts.❤

[@DineshGudla]

@ankitbansal6 This was great video covering multiple technical stack of AWS. Simple yet insightful project for beginners

[@ankitbansal6]

Glad you liked it 😄

[@adharshsunny5154]

Wow Amazing content as usual

[@ankitbansal6]

Glad you think so!

[@baskaranpbs8906]

Awesome video and really helpful

[@ankitbansal6]

Glad it was helpful!

[@saikathazra6732]

Sir are you going to start any AWS course?

[@kolisettysasiram290]

Nice initiative bro I felt in the coming videos you can add some uses of Aws service we are using in the videos.. also some more usecases..

[@ankitbansal6]

Sure 👍

[@Mehtre108]

Hello sir, its like a incremental load instead full load .

[@ankitbansal6]

Yes, correct

[@Mehtre108]

@@ankitbansal6 can you make project on data source would be MySQL, SFTP. Like realtime.

[@nipunshetty9640]

Can u make FROM BEGINNING TO END PLAYLIST OF CLASSES OF MICROSOFT AZURE AND AMAZON AWS which we can utilise d same during work, please ankit bhansali my request for you @ankit bhansal

[@AnkitMinj-wc1pm]

Hello Ankit, Your content is very helpful. I was seeking clarification to understand the data source needed for connecting to Quicksight for creating visualizations, as explained in this particular video. Is the process used in AWS specific to a data engineer's skill set, or is it a must-know skill for data analyst roles as well? If so, please assist us in identifying the AWS topics we need to know in order to connect a data source to Quicksight.

[@heetshah5923]

On upcoming days can you make video that cover that redshift also. And if possible can you make fetch the data from redshift to powerBI process only

[@mohansai9109]

Please create a project on Azure data engineering project. Pyspark, Databricks and Azure data factory. By using this please creata a data engineering project

[@suresh2105]

Can you make a playlist on how to start with aws account for data analysis. Like stwp 1 account creation Redshift, lamda, s3 etc. Each service and uses foe data science. Please?

[@amanahmed6057]

Sir do we need account of AWS , can you share any virtual playground who is giving aws platform

[@sushilkumarsss]

Sir, please suggest, are you planning any courses on AWS? If yes, when will you start it?

[@bihaskumar5857]

Thank you for this video.. We can upload the csv file in a s3 location and using aws glue we can create the snapshot folder. Just a thought

[@ramanaswamy9845]

Do you have Any plan to launch AWS data engineering course?

[@hiteshjha8850]

I am unable to create an AWS GLUE .its saying access denied. I searched on google and tried every possible solution mentioned , but none of them are working

[@ankitbansal6]

If you have logged in as an iam user then make sure you have glue access.

[@hiteshjha8850]

@@ankitbansal6 yes I tried both from iam user with glue access as well as from the root user

[@shivanandkumar6002]

Thnkss alot for this Ankit bro 😎

[@manjumohan7731]

Can you also create E2E Data Engg Projects using Azure

[@rakeshkumarsharma2250]

Bro I am following this channel since 1 years , you are doing awesome work and want to learn aws cloud for mlops suggest me good sources thanks

[@vivekchandan3441]

This is really a great video. Please make more by taking bigger data & show us the lifecycle like end to end project on data analysis.

[@baskarandurai6702]

Thanks for your efforts Ankit... Please continue this alonn with python problem solving video like SQL.. Waiting for python Ankit..

[@entertainmenttv1672]

Please make more videos like this, also try to include pyspark as well.

[@SamyukthaYanapu]

Hi Ankit, Thanks for creating AWS informative content that makes complex topics easy to understand. Much appreciated! Expecting more such videos..

[@abhishekkumar-qi3is]

Sir please bring aws data engineering course also in live mode.thanks

[@ArvindPatel-ht3cp]

This is so amazing for someone to excel in data engineering!!

[@khaqanazam-p9v]

Thanks, Ankit for sharing this AWS project. Please could you tell me how we can schedule the dashboards on quicksight like when the data will be updated in S3 bucket dashboard automatically should be updated? Or is it possible

[@ankitbansal6]

You can schedule quick sight dataset refresh in quick sight only

[@abhijeetjain8228]

Sir, please continue more on this series thanks.

[@partharora6023]

sir how to get job after 3 years of gap

[@rishavsen5581]

This helped pave a way for one of my existing projects, where I was facing a blocker. Thank you Ankit

[@ankitbansal6]

Great 👍

[@ibizilong8907]

please do one video with ETL as well

[@heetshah5923]

This will include redshift?

[@electricalsir]

Thanks man we are waiting for this video ❤😂

[@zainaltaf4935]

great learning source. Thanks for it

[@MsMohanj]

When I worked at Anz, I saw you and became a member of your Python and SQL batch. I am your biggest admirer and follower. For all data engineering careers, your channel is the place to go. I'm grateful for your efforts 🎉🎉🎉🎉

[@ankitbansal6]

Awesome, thank you! 🙂

[@hasnainakbar8621]

❤

[@griffingalatta3619]

I'm a Talend developer.. I'm shifting my Career as AWS data engineering....this project looks simple..will the real time company data engineering pipeline will be like this onlyyy

[@ankitbansal6]

It started with simple. Will do more complex projects 😊

[@griffingalatta3619]

thanks looking forward to it @@ankitbansal6

[@AbhishekJain-wl3st]

@ankitbansal6 I am getting access denied error while creating crawler can you please help me in resolving this issue

[@priyaghosh9971]

Hy , is your issue resolved ?

[@BALAJI-je1ty]

Thank you very much

[@ravitejatavva7396]

Great Initiative, Keep going @ankit..😍

[@ankitbansal6]

I will try my best

[@vikramjitsingh6769]

Good One!!

[@aur505nj]

Thanks Ankit

[@mubashiraf2524]

Is these simple data engineer project?

[@ankitbansal6]

Yes

[@abhishekn786]

Great initiative, thankyou ❤

[@hritikapal683]

That's a great initiative sir, keep them coming!🎉

[@ankitbansal6]

More to come!

[@mohitjoshi8984]

Thank you bro appreciate your efforts 🙂

[@ankitbansal6]

It's my pleasure

[@yashsaxena7139]

How much will this cost to run on AWS

[@ankitbansal6]

less than 1 dollar.

[@arshS27]

When is sql live starting end to end ?

[@ankitbansal6]

March 2nd

[@arshS27]

@@ankitbansal6 thanks Ankit ! And do u also cover tableau training and python as well as I want to learn this trio ?