YouTube Data Analysis | END TO END DATA ENGINEERING PROJECT | Part 2

[@aman_nv]

this is great data engineering project by the way and well explained, especially for beginner who starting to learn.. i have learnt a lot. thanks, Darshil. those who are stuck at creating AWS Glue ETL job, seems AWS updated the UI and settings of it. it's not the same way as he shown in the video, there is another way you can do that, -->he attached the PySpark code for the ETL job creation in the description, ---> then, AWS Glue --> ETL jobs --> choose option create job using script , now simply copy script in there. thats all. save the job and try running it. it should work.. Note: Make sure the data source details are modified as per your database name and table name in the script, other wise job may fail with an error Thanks!!

[@KatyayaniDatti]

@aman_nv I too have completed the project over this weekend. cheers to us 🥳 Coming to the ETL Job creation part, copy pasting the script is fine but as a beginner I find it difficult to write such script. The old UI was better where it generated the script for us based on our actions, do we have something similar in the new UI?

[@iamayuv]

00:03 Creating a crawler to understand and analyze data stored in AWS S3 buckets 05:48 Query execution and data type casting 11:43 Preprocessing and Efficiency for Querying 17:16 Writing data into the target bucket and creating partitions 22:05 Create a glue crawler to clean and catalog data 27:25 Data processing pipeline created using AWS Glue Studio 32:23 Created an analytical pipeline using AWS Glue to transform and store data 37:18 Building reporting version of data makes it easier for data scientists to analyze and query the data. 42:01 Create a dashboard to visualize data from KZbin

[@mrbcan7215]

Hi, Can you explain me how the "raw_statistic" table has been created autamatcally after he created the crawl_1, When I tried same processor, it didint work on me

[@robertmoncriefglockrock8957]

This is a simple error I ran into gonna post it here incase others have the same. When trying to run the job @21:25 I was getting "NameError: name 'gluecontext' is not defined. When adding the line "df_final_output = DynamicFrame.fromDF(datasink1, gluecontext, "df_final_output")" I accidentally forgot to capitalize glueContext, instead I put gluecontext Thank you for this walkthrough, I start my new Data Engineering job tomorrow and the company uses AWS so this has helped me tremendously. You are doing magic my friend

[@vishnuvardhan9082]

Hi Robert, Hope you are doing well. it's been over a year since you posted and joined your new company. Just wanted to check if this new job was your first data engineering job or if you were already experienced in DE? And how are things at your new workplace?

[@LaxmikantMaheshjiKabra]

Great work Darshil! I have only 1 suggestion after finishing the whole project along with the video which took me around a total of 6-8 hours except the dashboard. My suggestion is that you can take a minute extra and explain the code properly so that we viewers can understand what transform actions are we taking in the ETL because that would make more sense to the video overall and why you chose the steps were there before and after ETL step becomes clearer. Though, thanks for this wonderful project and I am probably moving to the Azure analytics project after this one.

[@Ujwalarao]

Thanks for bringing me close to the real use case scenario of Data Engineering.

[@freelychanu2086]

Well explained everything 0 to 100% , thank you so much!❤👍

[@lâmtrịnh-r4m]

12:08 if u still got issue with hive_bad_data after following step by step : u need to convert id to bigint by lambda funtion : add this code at line 21 " # Extract required columns: df_step_1 = pd.json_normalize(df_raw['items']) df_step_1['id']=df_step_1['id'].astype(int)"

[@VishalThota-p1r]

Appreciated your efforts Darshil

[@rahulgyani2965]

Hi Darshil, Thank you for this video. I have a question for you, when you have created a cleaned version for csv to parquet, why didnt we use lambda function instead of glue job?

[@youraverageguide]

Darshil is a great teacher! Great project.

[@PriyankaGopi-j6m]

Finally completed this project. Thankyou so much for this! You're a gem :)

[@SatishSharma-rh4su]

Heyy, I am facing an error while joining cleaned and raw data. Can you please help?

[@sreyassawant2205]

Hey I am facing same issue. Can you help out?

[@PranavPranav-w9i]

how did you created job in glue i am not able to create there is no option for it

[@assieneolivier5560]

Finally get this project done. Great project to learn data engineering!!

[@chinmaymaganur7133]

Hi how did you set up the etl job?

[@xandao30]

@@chinmaymaganur7133 good question

[@ajtam05]

Another great video. Only thing is...AWS has updated Glue console along w/ other consoles. I believe I updated accordingly, except for the schema datatypes (which it looks like I change update after the job is run). But for the script...it does look entirely different. Could you assist w/ an updated vid on using the new Glue consoles?

[@jenithmehta9603]

I am facing the same issue

[@ajtam05]

@Jenith Mehta If you scroll to the bottom of the navigation pane there is "LEGACY" versions. I realized after I posted this but I used that. Hope that helps. 😀

[@rohanchoudhary672]

@@ajtam05 can't find that

[@SCREENERA]

Finally,after 100times of disappointments...i done it...Great Efforts and it's my very first Project in DataEngineering field...Thanks ...Errors are challenging but only whose who have a real interest in DataEngineering.He will definitely achieve it ..by.Done this project completely.

[@N12SR48SLC]

An error occurred while calling o88.getDynamicFrame. User's pushdown predicate: region in ('ca','gb','us') can not be resolved against partition columns: [] in my job 23:00

[@SCREENERA]

@@N12SR48SLC sorry bro.

[@SCREENERA]

@@N12SR48SLC yes bro

[@SCREENERA]

Don't forget close the activated services from AWS..after project got done

[@SCREENERA]

@@N12SR48SLC what's the error

[@dsilvera1578]

Darshil I learned alot. I believe this is helping many persons. Thanks for all the effort you put into this.

[@srinivasn4510]

best project from scratch Thanks bro☺☺

[@jsingh7810]

Thanks Darshil!! Finally made this cool project after overcoming all those errors. Really good explanation.

[@GustavoFringe-dv2yg]

@@aishwaryapatel7045facing same issue

[@lifefacts7368]

I have parquet file error when i do string to bigint , i also delete the file from s3 bucket but not working , any one can please help me .....

[@blanguss]

I am stuck at 15:00. After adding the node for the data source path node, I added a transform node, then the node for the target path. However, I cannot seem to find a way to edit the data types like he did (from long to bigint). I cannot find the interface to compare the schemas. The UI has changed a lot and I am struggling to figure it out. Someone help

[@nitishkaushik2076]

simple and to the point explanation. Great work bro 👍🏻

[@adib4361]

How do we showcase this project in our linkedin profile our in our resume

[@piyushpaikroy3579]

hey Darshil.... I hope the project is complete!!

[@rushikeshdarge6115]

Thank You for the Awesome Tutorial!!!

[@ajitagalawe8028]

Too good. Learned alot.Thank you

[@umerimran3833]

that was the awesome project, Thank you!

[@saitarun3246]

Great video Darshil, thank you so much!!

[@lifefacts7368]

I have parquet file error when i do string to bigint , i also delete the file from s3 bucket but not working , any one can please help me .....

[@harshitjoshi2250]

To the folks struggling with glue script to filter regions out: try deleting the region files manually from S3 (make sure to enable bucket versioning so that objects are not permanently deleted). By doing this you can check if rest of your code is good, and even go on with rest of the video if its working.

[@iamgdsclead4208]

I am getting timeout error

[@chayanshrangraj4298]

I think it's better if you just move the folders somewhere else so you won't have to upload them again in the future.

[@lguerrero17]

Which part of the video ?

[@snehakadam16]

Hey hi, can you please share the pyspark code?

[@akshayrajput9049]

great work darshil bro ...can you send the ppt if possible

[@neha4024]

Can you provide ETL script shown in the video. I am getting error even after adding predicate_pushdown

[@tanny_tales]

did you resolved it ? I am facing few issues in it kindly help

[@shutzzzzzz]

Do i have to pay anything to complete this project? Or it is completely free?

[@imenbenhassine9710]

@darshil thanks for the effort, great job!! I just finished the project and so proud of myself, my very first project switching from DA to DE. thanks a lot

[@ybalasaireddy1248]

Hey hi did you get lambda timeout error by any chance?

[@allenclement5672]

hey iam seeing the new aws glue ui. how did you create the job there? iam facing a lot of confusion on what to select and to navigate in that .the video ui is different.

[@vighneshbuddhivant8353]

@@allenclement5672 hey did you solve this issue

[@uditkapadia7104]

​@@ybalasaireddy1248if you are getting time error...pls increase time

[@KomilMustaev]

@@allenclement5672 same problems... Did u solve them? can u help me?

[@kuldeep_garg]

You are doing such a great work, please should learn from you how to teach by this learning by doing method… Please do some more projects like this using real time data, big data also so that we can learn that also. And thanks again this tutorial is helping a lot🎉❤

[@sharafmomen2460]

Really great project! Just wanted to ask, when more data ends up in the landing area, will the rest of the processes after automatically go through the pipeline you created? Because it seemed like some parts you had to do manually, like using AWS Lambda.

[@mkdTech369]

Great

[@TanmayMeda]

Thank you so much for the Amazing video

[@ashutoshdixit2049]

Good work ,it's a grate project ,helped out learning many things.

[@sanikaapatil7279]

can you help me idont understand with new interface of awsglue

[@vishalkamlapure3344]

Thank you Darshil for this wonderful project.. I have been looking for such project for long time.

[@ishan358]

How did you solve runtime error

[@chayanshrangraj4298]

@@ishan358 What kind of error are you getting?

[@lguerrero17]

​@@chayanshrangraj4298 can You help me with an error ? In the step of aws glue to do the join of the tables

[@chayanshrangraj4298]

@@lguerrero17 Sure! What is the error that you are facing?

[@lguerrero17]

@@chayanshrangraj4298 When I try to do create the etl to generate table analytics , it creates the table but doesn't generate columns and rows.

[@ajtam05]

Has anyone used ProjectPro before? I'm considering investing into it, but just wanted to see if anyone has experience with it yet? Looks promising.

[@nikhilrunku8877]

Hi Darshil, I have been trying to implement this project. At 13:28 you have created a job, but I am not able to see that option in the current version. All I can see is to create the ETL job visually. Can you please help me with this?

[@rushikeshdarge6115]

Did you get a solution?

[@tanny_tales]

@@rushikeshdarge6115 did you get a solution?

[@kushpatel699]

Hey, did it got resolved??

[@kushpatel699]

@@rushikeshdarge6115 Hey, did it got resolved??

[@rafdeo]

@@kushpatel699 proceed with creating it visually via drag and drop. s3 as source > transform schema > s3 as target. Only thing is for me the partitioned region column doesn't pull in so I am not able to do the next step. Hopefully you have better luck

[@atharvasankhe1153]

how to get the Jobs ( 13:30 ). Apparently, the Glue console has changed , so not sure how to go ahead

[@vineetchotaliya3978]

got solution to this ? 💀

[@GauravKhanna-cl7gd]

The Athena query works first time on the parquet file, and then I have to delete the unsaved folder in the cleansed bucket, has anyone dealt with this, I am still at the 5 min mark of this video. Really frustrating!!

[@prasannakusugal4333]

Thanks for the great video Darshil !!! Leant allot of new things :)

[@lifefacts7368]

I have parquet file error when i do string to bigint , i also delete the file from s3 bucket but not working , any one can please help me .....

[@subashpandey518]

someone plz help, the UI for creating the job completely changed. I am not able to create new jobs.

[@marvinarismendiz2814]

Here the same @Darshil Parmar

[@kopalsoni4780]

Hey all, I am stuck at 40:35. I don't see the Database option for 'New Athena data source'. Not sure if QuickSight had an update since this video was created. Any suggestions?

[@kopalsoni4780]

Answering my own question, had to change the region which was a default selection.

[@shutzzzzzz]

Thanks. Can u plz tell me whether u need to pay anything to complete this project?

[@dimensionalcookie]

thank you so much

[@jolaoduwole4523]

I'm stucked at 12:55 I'm unable to go pass the id type error... I deleted parquet several times but still not working

[@divyakhiani1116]

I did everything in us-west-1 (California) region, but this region is not available in quicksight. Can you help please @@kopalsoni4780

[@BobbyCambos]

It seems like for me at the 28:26 the parquet files didn't transformed. I checked the trigger and the region but still no finding solution. Do anyone has any idea ?

[@이신우-x3m]

Did you remove extra white space in prefix and retry it? I solve the same problem in this way.

[@BobbyCambos]

@@이신우-x3m yes, and i had the same result. Only blank database with only the column names and no parquet file uploaded.

[@ahmedopeyemi2980]

Thank you so much@@이신우-x3m​ . After days of asking, I finally found a solution.

[@Trendy-Bazar]

Hi, I am getting error while creating Glue ETL job 17:00 the UI is completely different and cannot proceed further, any help?

[@ChandanaandSanthosh]

same here.. stuck there

[@srihariraman9409]

@@ChandanaandSanthosh ive set the job pipeline using the new UI, but the script editing is mismatched

[@saiganesh5702]

hey parakh did your issue with ETL got resolved? if yes can you please help me with that

[@yashiyengar7366]

Same for me stuck at ETL job creating section

[@dhruvingandhi1114]

@@srihariraman9409 how did u set the new UI pipeline Please mention few steps

[@madmonk0]

Is there an updated version of this? The Legacy Glue UI cannot be accessed now

[@florenceofori7930]

Iwas also searching for it. I'm wondering what to use now.

[@kushpatel699]

Hey, did it got resolved??

[@kushpatel699]

@@florenceofori7930 Hey, did it got resolved??

[@rajrockzz3797]

Great video bro..

[@meenachir1167]

Hey.. How to convert from csv to parquet for other regions like Russia,Korea etc..

[@pawandeore1656]

informative

[@kushpatel699]

Hello @Darshil, thank you for this informative project. I am following your project meticulously, however, for "Add Job", AWS says this "This page has been removed from AWS Glue.". How can I get the job added. Thank you.

[@krishanwannigama3450]

Anyone who is struggling with trigger ,please make the trigger in s3 bucket . That will work perfectly

[@akashpandey7034]

Will the AWS will charge us for using these services shown in the video?

[@shivakrishna1743]

Thanks, for this!

[@observerXIII]

why did you re-create the crawler at the start of the video?

[@PranavPranav-w9i]

How to create new job this option is not there in my glue

[@dushii_life]

The interface is changed completely i am still figuring out what's the problem.

[@Lapookie]

INTERESTING POINT HERE : at 4:56 How can you know what primary key to choose to do the INNER JOIN ? Before watchin i tried to make a.video_id = b.id Because it sounds logic that each row is unique, so the video_id should be used and compared to the id of the other table that are also unique video row. Am i wrong ? Anyone have idea ? Thanks a lot

[@avni_103]

Go and read data columns description for that While doing such type of join u will be given proper knowledge of data

[@prafulbs7216]

S3 rigger is not working for me, I tried many times. The data is not writing into s3 cleansed bucket(json files)

[@eduhilfe1886]

Please check is there any space while defining Prefix: youtube/raw_statistics_reference_data/ . If you are coping from s3 then there may be some space after youtube/

[@harshalshende69]

hello brother , have u find the solution i m getting same error

[@prafulbs7216]

@@harshalshende69 i actually did manual work, by uploading the files.

[@robertmoncriefglockrock8957]

@@eduhilfe1886 This fixed it for me thank You!!

[@robertmoncriefglockrock8957]

@@harshalshende69 Check your s3 trigger, make sure KZbin/ doesn't ave a space after it

[@satishmajji481]

@Darshil Parmar - "region=us/" folder is not created for me; only ca and gb folders are created upon running the ETL job. PS: I added "predicate_pushdown = "region in ('ca','gb','us')" as well but floder is missing for "us" region. Can you please take a look at this?

[@pdubocho]

Same thing happened to me. Error occurred when initially using AWS CLI to load data into s3 buckets. After executing the command to upload the csv files, I did not hit enter after the upload was "complete". I just exited the cmd box. To fix I manually uploaded the data and re-ran processes from both videos edit: this is only valid if you go into your raw data s3 folder and don't find the folder "region=us"

[@jarlezio2463]

because us is not present in the initial dataset

[@Soulfulreader786]

use aws cli for creating folders using cp command

[@tanny_tales]

After adding predicat pushdown i am still getting error UNCLASSIFIED_ERROR; An error occurred while calling o116.schema. Unable to parse file: RUvideos.csv

[@zaheeruddinbaber6762]

Hi, where to get the ppt you are using?

[@surya-z9e]

hey bro, your videos was very understandable. could you make the video deeply about the quick sight.

[@YuyuZhang-v3q]

Hey is there someone can help me? The UI of ETL Jobs has changed a lot and I cannot add a job successfully.

[@chinmaymaganur7133]

were you able to figure this out

[@gabilinguas]

Hello! I faced this issue and I figure it out. You go to ETL Jobs, click on the button "create job from a blanck graph" and go to "job details" on the menu third item

[@gabilinguas]

The second part, when it clicks next, you have to go to the visual (first item on the same tab you clicked on job details), add a node, first you choose the S3 bucket to choose the source, than next you add a new node from schema, and than a third node on the target tab

[@RonitSagar]

Hey ​@@gabilinguas I am not able to get what you said in the last comment. Can you please little more😊

[@gabilinguas]

Hey @@RonitSagar ! You can follow basically the same steps described in the "Build ETL Pipeline" in 30:33 on the video. The process is almost the same, you just have to pay attention on the different details.

[@N12SR48SLC]

not able to see region column in my schema, also all columns showing string as the datatype

[@priyadarshinibal9304]

sameee. did you find any solution?

[@vallabhajoshulakrishnachai6475]

@@priyadarshinibal9304 same here did you find the solution?

[@SCREENERA]

Thanks a lot Darshil and project pro

[@N12SR48SLC]

not able to see region column in my schema, also all columns showing string as the datatype 16:07

[@AryanDurge1]

is the athena serivce free or do we need to pay for the usage

[@AryanDurge1]

do we need to pay for the useage of the aws athena service ??

[@dushii_life]

No it's free for new yusers for one year

[@jatin7089]

I am stuck on creating Glue job as UI is different. Please anyone help here . Where to change data types. I am able to add source and target

[@shubhamnikam4759]

stuck in same I added data target and source but not able to figure out how to change data type

[@rohitmalviya8607]

@@shubhamnikam4759 @use google gemini

[@ChaosChuckler]

How to build this project through IaaC approach? So that i can delete it to avoid AWS charges and re build when necessary

[@pankajchandel1000]

is there a project where you used python notebooks or emr for processing data instead of lambda functions ?

[@prafulbs7216]

Add trigger to lambda function, not working for me. Tried many times, Please suggest.

[@divyakhiani1116]

facing same issue. Did you find a solution ?

[@prafulbs7216]

@@divyakhiani1116 I redid the same steps once again, I guess. I don't remember, though !

[@ShaunDePonte]

You didn't answer the initial question as in video 1: How to categorise videos, based n their comments and stats and what factors affect how popular a youtube video will be

[@yaryna_ch]

So what we should do with data like japanese?

[@Soulfulreader786]

BEFORE TRIGGER DI D YOU CHANGE LAMBDA TO TAKE ALL RECORDS..LIKE INITIALLY IT WAS [ "RECORDS"][0]

[@jenithmehta9603]

Job creation UI has completly changed. I am stuck at that step.

[@russophile9874]

go to the script tab and click edit. Paste the spark code from the githup repo, it will work.

[@subashpandey518]

@@russophile9874 could you please explain precisely what script tab and which edit? I am stuck on this step. thanks

[@manigowdas7781]

Just completed this project. Thanks for the Content , understanding AWS services and using them for our use case is really crazy thing! @DarshilParmar ❤ #AWS CLI #S3 #Lambda #Glue #Crawler #Glue Studio #Glue ETL #Athena #Database #Quicksight

[@saiganesh5702]

hey manoj can you please help me with new etl job visual editor scripts I am facing trouble to understand that

[@yashiyengar7366]

@@saiganesh5702 even I am facing issues in ETL job creation section due to new UI

[@chinmaymaganur7133]

how did you set up etl glue job

[@chinmaymaganur7133]

@@saiganesh5702 were you able to figure this out

[@danielofuokwu6595]

I can’t visualize the data on quick sight it is says i don’t have permission

[@vasanthkumar8120]

Hey thanks for this great video. I want to know, how much does it cost to complete this entire project on AWS?

[@AngelYaelRodriguezGarcia]

do anyone had the same problem with the job interface change?

[@dushii_life]

Mee too

[@herdata_eo4492]

@projectpro pls consider monthly subs instead. billed 6 mths/yearly is too much.

[@ProjectProDataScienceProjects]

Hey, we have some discounts going on and it's valid only for a few days, please share your email id and our team will get in touch with you. thanks

[@ueeabhishekkrsahu]

Where is the discord link?

[@kaushiksarmah999]

Hello Sir, I m not able to convert the id field type to bigint i tried the steps as according to the video multiple times. Even looked online for the procedure but got noting as such. Can you help me sir?

[@deepakpradhan9743]

Has your issue resolved converting id cloumn to bigint

[@yeezco2535]

No

[@nguyentiensu4088]

When my lambda function is triggered by an S3 event, the cleaned_statistics_reference_data table is created. But when I check by SQL command "SELECT * FROM cleaned_statistics_reference_data", the result is an empty table. I tested the lambda function with a test event, and everything is OK (there is data in the cleaned_statistics_reference_data table). Please help me with a solution! Thank you!

[@drishtihingar2160]

have you found the solution I am facing the same issue. Please help me

[@nandinisingh9217]

@@drishtihingar2160 facing the same problem have found any solution?

[@drishtihingar2160]

no not yet@@nandinisingh9217

[@rohitmalviya8607]

you have to upload json files through cli AFTER creating the trigger.. lambda wont process already existing json files

[@SohamKatkar-xw5uf]

​@@rohitmalviya8607 Hi, can you tell exactly which json files need to be uploaded and where to upload them ?

[@shantanuumrani9163]

In 16:54, I'm not able to see the region source key in my output schema. What should I do?

[@-aadfi-1710]

what is the solution to this?

[@SCREENERA]

Don't forget close the activated services from AWS..

[@sivasahoo6980]

Can you tell how to close it We have to delete bucket and etl job or something else

[@SCREENERA]

@@sivasahoo6980 Delete all the services

[@TheAINoobxoxo]

what to do about creating jobs @darshit #darshil should i use the script that is given as now aws has moved to visual etl and simple job creation has became complex for someone who doesnt know how to work with visual etl

[@banarasi91]

why my trigger is not invoked when file is uploaded in s3 ,although my test is properly working in lamda function,it is not showing any error also. i am not able to understand the issue

[@sivasahoo6980]

did you get any solution

[@banarasi91]

@@sivasahoo6980 its been time since i posted but if i remember correctly it was somewhere with naming or syntax where there was extra space which i was not able to find then rewatcing evrything i got it, idont exactly remember where but may be in some path

[@banarasi91]

hello guys,you might be getting error at the point of testing that is because of db name has been not changed in environment variable, please take care he has forget to change db name , if you notice in athena database name is db_youtube_cleaned but it should be de_youtube_cleaned, which is giving error in lamda final testing as "Entity not found"

[@sivasahoo6980]

@@banarasi91 thanks a lot yeah there is a extra space in path

[@N12SR48SLC]

@@banarasi91 not able to see region column in my schema, also all columns showing string as the datatype 16:07. my etl job is also failing

[@vijayarana8208]

Hello, Darshil I am kind of stuck at(22.52) of the video. My job runs successfully but the raw_statistics folder is not created. I have described the region correctly in the code. any suggestion would be helpful ,

[@anupammathur918]

Check the s3 trigger remove the space after youtube/

[@ajitagalawe8028]

caught with the same issue. In my case , files were created directly in raw_statistics. There are no sub-folders "region/". Could you please help me? Thanks

[@ueeabhishekkrsahu]

Can you please share your script? I have created my job but it is not executing. Please share, it will be a great help.

[@officialPatel-e3q]

Actually in my case i am getting confused in creating the job because in the current aws ui it directly shows visual etl there is no option of target and data transform and no option of adding a job manually..if anyone could please help me with that

[@ganeshb.v.s1679]

Hi I am in the last step of building ETL pipeline. I successfully created the glue job named 'de-on-youtube-parquet-analytics-version' . The contents in the de-on-youtube-analytics bucket are getting added but there is no creation of 'final_analytics' table happening. Please help me resolve the issue. Thanks in advance

[@086_AASTHASHUKLA]

hi ,I created the glue job but it isnt creating the same files under raw_statistics as shown in the video how did you do it

[@NikitaRamakantChaudhari]

hi, did you get the solution for your issue

[@ebubeonuegbu3467]

I added this code: predicate_pushdown = "region in ('ca','gb','us')" and got this error "Error Category: UNCLASSIFIED_ERROR; An error occurred while calling o103.getDynamicFrame. User's pushdown predicate: region in ('ca','gb','us') can not be resolved against partition columns: []" the source S3 data in my set up is partitioned by the "region" column. Please how do I resolve this?

[@iamayuv]

Bro have you found the solution ?

[@FRUXT]

Thanks ! Why parket file ? Is not it more simple to keep everything in json or csv ?

[@marvinarismendiz2814]

think it is cause the larger volumes of data, can use spark

[@isaacodeh]

I did asked you a question on your channel about the wrangler which didn’t seems to be working for me. I don’t know if it has to do with location?

[@DarshilParmar]

Yes, it is only available in some locations

[@isaacodeh]

@@DarshilParmar oh I see! Thanks for the work you do!! You have been very helpful!!!

[@Lapookie]

@@DarshilParmar why is that ?

[@DeepUpadhyay-gs1ky]

where is the link for discoed ?

[@vishwajithsubhash6269]

I understand why we need to convert JSON to parquet, but why do we convert CSV to parquet it's already clean right?

[@vineetsrivastava4906]

parquet file format is more optimized and is faster- read more about it on internet

[@sanikaapatil7279]

Actually in my case i am getting confused in creating the job because in the current aws ui it directly shows visual et/ there is no option of target and data transform and no option of adding a job manually.if anyone could please help me with that ​@@vineetsrivastava4906

[@jayitankar7919]

Hi my Lamda triger for json files is not getting fired dont know whats wrong .

[@aneeqbokhari4611]

Yeah same. Have you figured it out?

[@robertmoncriefglockrock8957]

@@aneeqbokhari4611 Same here

[@bukunmiadebanjo9684]

Had to stop here too. After deleting all files and re-uploading, trigger does nothing.

[@shantanuumrani9163]

​@@bukunmiadebanjo9684 The same thing happened with me. And one figured out to solve it?

[@bukunmiadebanjo9684]

@@shantanuumrani9163 didn't find a solution. The whole UI also looks different as AWS already made changes so I decided to move to a different course and abandoned this.

[@merkarii]

You go so fast

[@lifefacts7368]

I have parquet file error when i do string to bigint , i also delete the file from s3 bucket but not working , any one can please help me .....

[@kratos_gow610]

13:39

[@venkatsaiphanindraanagam]

we created ETL job to join data so that when new data gets added to the bucket it will be automatically joined instead of running an SQL query. But shoudnt we trigger this ETL job for the data addition event in S3 ? Can anyone answer this

[@bhumikalalchandani321]

No, i think only 1 time lambda trigger from s3 happens for .json file to paruqet --> then cleasend s3 bucket if filled -> from there analystics data picked.. confirm this

[@merkarii]

But good work

[@Sdsatya]

Excellent !!

[@Lapookie]

Thank you a lot for this project! It helps me to understand what tools we generally use as Data Engineer to build data pipelines etc. But, I don't feel like to have learned how to do it myself. I mean, i have followed you along, and understand what we made but, I need more explanations on how you process the data, how you get your bucket with AWS Lambda (the code is not explicit to get when doing this " bucket = event['Records'][0]['s3']['bucket']['name'] key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')") Need exercises myself

[@mananyadav6401]

You can go through the test event that we generated. There is a json in the test event that we are using to test the function. Try to navigate that , you will get the understanding how bucket name is captured etc. hope it helps

[@Lapookie]

@@mananyadav6401 Oh yeah, i'll do that, good idea, thanks for your answer :) !

[@RizwanAnsari-lt3nf]

Is there any one facing issue with the lambda function as when I have added the trigger but the nothing new file is created once i upload the json file to the same bucket.

[@tanny_tales]

yes yes same,did you got the solution

[@cryofficial2873]

bro did you got the solution