End-To-End Data Engineering Project in 40 Minutes | AWS Cloud | PySpark

[@shivam87480]

used aws glue under free teir cost a some amount after i month which i used because i was unaware of the extra charges..so i request u please provide a commet while making the video that which services can cost money and which services can be used under the free teir , that will be very helpfull for newbies like me...

[@hlulaniwinners7076]

That was really good to follow...100% worked and I learned so much more in 40min😀😃

[@adityatomar9820]

i got a bill of 2.80 dollars by just running glue etl once...I dont know how im gonna create more projects if they keep billing like thiss..i cant afford fee rn...? what can i do?

[@BOSS-AI-20]

@@adityatomar9820 make new free tier account

[@OlafKoch-j5y]

thats what i was looking for. thank you :)

[@OlafKoch-j5y]

also, you should create a playlist with all data engineering projects you already done, gonna be easy to find :)

[@ravi19900]

Amazing content... This is first AWS DE video I watched in practical and I am glad I found this video. Thank You Can you please share some automated way of doing ingestion process in s3 staging folder and some preprocessing demo followed up by some SCD Type 2 implementation on glue?

[@TonyRydinger-bq9pk]

Great video, can you please explain the preprocessing part, what exactly did you use to preprocess the datasets, was it a python script in pandas or something else?

[@dggh7879]

Great project for beginners!!

[@ganesh.majety5260]

Just watched 1 video, u gained a subscriber 🎉. Hope more from u😊

[@rampatil-t6u]

Incredible work on data engineering project! The ability to design, implement, and optimize the entire pipeline from data ingestion to processing and visualization is a true testament to your skillset. The attention to detail, efficiency in workflow, and the seamless integration of various tools and technologies are impressive. Your understanding of data architecture and best practices shines through in every step of the project. Keep up the fantastic work

[@datewithdata123]

Thank you very much!

[@adityatomar9820]

OH GOD! AWS UI always made me overwhelmed and scared me....But you just explained everything so beautifully...Thankyou soo much mann.....I finally feel confident that i can learn AWS and build awesome projects... BTW will AWS charge us for using ATHENA and GLUE as they don't come under free trial...?

[@datewithdata123]

Yes. For completing this project the bill will be less than half a dollar(if you don’t run a glue job a lot).

[@sanjeevpandey2753]

Nice one bhai, very precise and clear explanation

[@datewithdata123]

Glad you like it

[@sanjeevpandey2753]

May I have your mail id please?

[@datewithdata123]

[e-mail]

[@avinash7003]

can you do on s3,glue,emr,lambda,athena,redshift

[@datewithdata123]

Ongoing. Will be released soon

[@swapnilgaikwad9738]

Good please again one end to end aws-data project video

[@BommineaniSai]

I'm facing an issue with joining the Tracks with the album & artist as it showing as NO SOURCE KEY for the album& artist join condition, Can you help pls?

[@bullshere1122]

same issue please help anybodyyyyyyyyyyyyyyyyyyyyyyyyy

[@datewithdata123]

Please look if you have provided correct join condition.

[@binod8720]

The crawler failed to automatically create the table from s3 directory where parquet datasets are stored, not sure what happened as i followed the exacts steps given the glue have been given access to s3 creating a role to that specific user, any feedback on how to resolve this issue?

[@manojk1494]

Are you able to resolve this issue ?

[@binod8720]

@manojk1494 yep resolved, thanks 😊

[@ibrahimfadhili6621]

I love it ❤Thanks man

[@datewithdata123]

I'm glad you like it

[@ajtam05]

Would you know why the 'Data preview' on joins may not populate any data aka 'No data to display'? I did a sanity check and the albums and artists files (in excel) , do indeed have matching data in the artist_id (album) to id (artist). But when I join on those conditions, as you did, it doesn't populate any data. Just to see, I tried right and left join, and that actually populated data for each respective side (oddly enough). Seems like a glitch, but because the script it simple and the join script looks correct. Do you know if the data types are converted or something else occurs behind the scenes when you join in Visual ETL?

[@ajtam05]

I basically can't do the project because the subsequent nodes require data being fed from previous nodes. But there's no data at the first join (album/artist). Really odd.

[@datewithdata123]

Please check you have your data in s3.

[@himanshusaini011]

Yes we do have the data in s3 but the same issue is also popup for me as well

[@danielpequeno33]

I have the same problem, could any of you solve it? @himanshusaini @ajtam05

[@mushkarasaiprakash1915]

how did you preprocess data, what all you removed or changed while preprocessing the data

[@vishalkanvajiya-j8t]

bro visualization ka bhi explain kro

[@gnaneshwaripanthagani3515]

In AWS glue when I am creating pipeline in transform join I am not getting option to select any source key can u plzz help

[@FredRohn]

I used infer schema, and that seemed to fix the problem for me :)

[@gnaneshwaripanthagani3515]

⁠@@FredRohnThank you so much,it works for me

[@kunalnkalore]

in real time do we have to perform these task regarding IAM, etc or do we have to jst run terraform scripts or something similar and our architecture or cluster spins up? can you clear this real time working process?

[@djsamxgaming5732]

I dont know why i am not able to see the output in datawarehouse, but i can see 100% success rate in job monitoring window. Could you tell me what will be the problem in this???

[@TonyRydinger-bq9pk]

Great video, can you let us know what did you use for preprocessing, was it a python script in pandas or something else?

[@tulasipanthagani6401]

can u please help y crawler is not running, it is asking some permission ,which permission we need to add

[@vidhyabharathi3947]

I am unable to run athena query it is showing unable to kind parquet format

[@datewithdata123]

Have a look if you have provide correct path for s3, with right permissions

[@kukhwa]

I've followed all the steps you shared, but I can't run the crawler. It seems like the error, AccessDeniedException, is related to CloudWatch logs. So, I've added CloudWatch logs full access. But still not working. Do you have any insights?

[@shaikgouse4u]

Cuz I have fixed this one and created table successfully

[@datewithdata123]

Please provide Iam permission to the user (administrator or necessary permissions)

[@vidhyabharathi3947]

​@@datewithdata123Even I faced the same issue i have provided full access but still unable to run the crawler

[@hemanthkumar7782]

@@vidhyabharathi3947 I have added glue service role and then it worked.

[@shaikgouse4u]

​@@vidhyabharathi3947 Try following below steps: Fix for crawler error: 1. Using Root user go into "AWS Glue console --> Getting Started page" 2. Click on "Setup roles & users" option 3. Choose your IAM User 4. Next stage select "Grant full access to Amazon S3" --> "Read and write" 5. Select the recommended "AWSGlueServiceRole" 6. Review & apply changes 7. Go to IAM console --> Access Management --> Roles. Here you'll see the role "AWSGlueServiceRole" created and assigned to IAM User selected in step-3 8. Re-run the crawler job and it'll complete successfully.

[@ruben3815]

good job dude!

[@adityatomar9820]

plz also tell how to push these kind of projects on GITHUB

[@nguyentien4711]

this procedure should not be on your github, it's just a BI tool while github is the place to show your code skill and project build merely by code from scratch

[@CricketLover-qy9nn]

I'm unable to the trackid from the join album and artist. What might be the reason

[@KomalChavan-ht7wm]

same

[@KomalChavan-ht7wm]

hey how u resolved this issue?

[@FredRohn]

use infer schema, that fixed the problem for me@@KomalChavan-ht7wm

[@FredRohn]

try infer schema, that made it work for me

[@danielpequeno33]

did you find a way to solve it?

[@eugenia6490]

Question please. 26min:38sec timestamp - you mentioned that the job created multiple blocks. Why are there multiple blocks? Thank you!

[@datewithdata123]

We have created two worker nodes and since we have very little data. we could see that there were exactly 2 files in our warehouse table.

[@eugenia6490]

@@datewithdata123 Thank you!

[@mkdTech369]

More Video please

[@Divya-gn5lh]

hey @datewithdata firstly I like ur project playlist if uhh share the source code with us it would be helpful for us.....thank for content

[@kumarsumit6117]

could you please help me after sucessfully running Glue pipline data s not stored in final s3 bucket

[@datewithdata123]

Please share your error SC at [e-mail]

[@rahulcp7013]

Were you able to resolve this issue, I am also facing the same

[@mwanthidaniel1254]

Is S3 a data warehouse or data lake?

[@datewithdata123]

S3 is neither a warehouse nor a data lake; it's an object storage service provided by AWS, but can be used as both because it can manage large volumes of structured and unstructured data for analytics, processing, and other purposes.

[@shivam87480]

can anyone tell how to showcase the project in github or put it in resume????

[@KomalChavan-ht7wm]

at time of trasforming enable to join table on condtion data is not fetching at column? is anybody help me

[@himanshusaini011]

Same issue with me

[@Gauravsingh-hx6lw]

When i add policy for glue its not working can you help me

[@AshutoshParashar-u5l]

glue_s3_role which you have created assign glue access to it it will work!

[@udaykirankankanala3635]

When i am trying to save visual etl job it is showing me error as create job:access denied exception What is the policy we have to add in root account?

[@datewithdata123]

iam:PassRole

[@udaykirankankanala3635]

I am unable to find that policy in root account Please help me

[@datewithdata123]

Or provide iam full access.

[@FredRohn]

did you solve this issue? I am experiencing the same thing. @@udaykirankankanala3635

[@FredRohn]

how do i do this? I'm having a similar issue@@datewithdata123

[@badboy1585]

hello bro, the services you are used in this project are comes in free tier right ? or we have to pay

[@datewithdata123]

Some of the services are not under free tier. For completing this project the bill will be less than half a dollar(if you don’t run a glue job a lot).

[@adityatomar9820]

@@datewithdata123 i got 2.80 dollar bill just after running etl once in glue

[@SS-gv8kh]

@datewithdata123 when I am running glue job it's successful but ouput files are not created in s3. Did you or anyone face similar issue?

[@AshutoshParashar-u5l]

the visual ETL for every node are you seeing greed ticked if no ten the ETL process is not completed as per design. Make sure all the nodes are green then run it. I faced same error and have resolved and its working as expected.

[@rahulteja4849]

While joining the tables in visual etl, i could not add the condition as i could not look for colum names it is not showing me any columns

[@tokyochannel5684]

Solved?

[@vichitravirdwivedi]

Refresh it multiple times. it happened with me too

[@datewithdata123]

This may happen sometimes when you have slow internet connection. Bcz glue will read the schema from data present in S3. Hence the connection need to be set.

[@himanshusaini011]

​@@vichitravirdwivedi I already did it multiple times but no output

[@FredRohn]

try to use infer schema, all of the fields popped up for me after doing that. @@himanshusaini011

[@akshaypy4117]

Crawler will not run with just s3 full access as shown here right?

[@datewithdata123]

You may need to add IAM:Full Access if you are working as an IAMUser

[@sidharthv1060]

@@datewithdata123 I have added IAM:Full Access also within the role glue_access_s3 but again failed to run crawler.

[@VivekYadav-og4lt]

⁠@@sidharthv1060I think you need add AWSGlue service role

[@supriya9047]

@@sidharthv1060 I am also facing the same issue repeatedly, even after providing all the required access.

[@kshitijjoshi2092]

@@supriya9047same

[@ajtam05]

iam:PassRole error when trying to attach the role to the project. iam:PassRole looks very confusing, but I'm not sure why no one else is encountering this issue.

[@ajtam05]

User: arn:aws:iam::905418287400:user/proj is not authorized to perform: iam:PassRole on resource: arn:aws:iam::905418287400:role/glue_access_s3 because no identity-based policy allows the iam:PassRole action

[@datewithdata123]

In the beginning while creating IAM user, plz add IAMFullAccess. This is happening because the "iam:PassRole" action is required when a service like AWS Glue needs to pass a role to another AWS service.

[@ajtam05]

@datewithdata123 OK, I will try that. I tried multiple solutions with regards to creating a new policy and attaching it to the user, but no luck. Hope that works. 🙏

[@ajtam05]

​@@datewithdata123 Yep, that worked. Thanks for that.

[@ajtam05]

@@datewithdata123 I believe that change has affected the way joins are occurring. before i was able to join the album & artist join w/ the tracks. but now the ablum & artist join doesn't populate any data. it looks like people have similar issue when i google, but no solutions provide online. are you aware?

[@vivekpawar3069]

sit please attach preprocessing of csv file code

[@ishwariupadhyay8122]

Can you provide your github link for preprocessing data.

[@datewithdata123]

Sorry didn’t save the code. We have used visual etl so the code was auto generated.

[@backgrounding4821]

Hello can you please update the Processed Data Link please.

[@datewithdata123]

drive.google.com/drive/folders/1PgZQDvw5GnvVQuhV7-MtxIZHnLsZA-Zs?usp=drive_link

[@backgrounding4821]

@@datewithdata123 thanks! (Y)

[@avinash7003]

please upload the Glue script

[@datewithdata123]

Sorry didn’t save the code. We have used visual etl so the code was auto generated.

[@HanhNguyen-sp8zo]

is it free ?

[@datewithdata123]

Yes