AWS Glue Tutorial for Beginners [FULL COURSE in 45 mins]

[@JohnnyChivers]

Hi Folks - The much requested update to this video with the new AWS Console UI for AWS Glue is now available on the channel with a new GitHub repo containing everything you need to follow along. kzbin.info/www/bejne/kKethJSfpLWMr9E.

[@sayakutube]

you were preparing the video when the entire west world was yelling "Happy New Year"! Great commitment and awesome result!

[@TerminatorAyan]

in 41 minutes, I never knew I can gain this much knowledge and at least be ready with AWS glue as we are in a transformation project. It really helped, thank you so much!

[@farhodotaboev3324]

If you had watched with 1.25 playback speed, it would take you 33 minutes

[@adeyemitunji343]

This the most detailed AWS glue video i have ever seen. Keep up the great work Johnny.

[@danielboza5747]

For me the most amazing thing is that he was working in the morning of January 1st. My respect! 😅

[@chamila.fernando.us2fernan663]

You are Awesome. watching in 2024... ETL steps needs minor updating but I was still able to follow ! Keep up the great work !

[@CaseWalker-j4g]

I am enjoying the video, thanks for this resource! I wanted to note that it seems you blurred your S3 buckets early in the video, but at 19:22 and also 21:33 when configuring glue, you do not blur your other S3 buckets. Also 24:49.

[@SungSam-wz2vk]

Awesome. I've never met people like you that you're so young with solid data science & aws skills.

[@JohnnyChivers]

Thanks Sung!

[@StephenGillie]

Troubleshooting tip for especially complex environments: If the user has access to ALL S3 locations in the table, then Glue will assemble the table for Athena to query. If even one S3 location can't be accessed by the user, then the table won't show up in Athena. Hope this helps complex schema lovers.

[@JohnnyChivers]

Good tip.

[@waeldimassi3355]

Amazing content by far !! Please continue with Glue. Never seen such high quality tutorial :D

[@JohnnyChivers]

Thanks for watching

[@manasisingh294]

you speak my favorite accent that's such a plus for me. Thank you so much for the quality content! :)

[@kakamoora7874]

simple and explain everything ... what we need .... thank you man ... i'm from sri lanka

[@jimaustin3608]

At 28:00 (AWS Glue Jobs section) current screens are much different than video. Figure all the same information has to be entered, but order and screen flips are totally different.

[@junaidraza6774]

I am not sure, does anyone said this before or not, but i wanted to say, You are a rock start.

[@vedprakash9413]

41 minutes was sufficient for me to get hands-on AWS glue. Thanks, Johhny for this awesome tutorial.

[@adalke2]

As someone transitioning from a data scientist to DE role, I found this extraordinarily helpful! Subscribed! Thank you!

[@arunsar7893]

I am interested in understanding why did you decide to make the transition.

[@JohnnyChivers]

Awesome! Thank you!

[@augustocarrillo5927]

I don't like subscribing to people but lit I wouldn't be able to learn aws without you. Best and unique content

[@EEdgerocks]

excellent Glue tutorial .. One of the best I have come across which teaches in a composed pace and easy to pickup for anybody

[@JohnnyChivers]

Thanks for watching!!!

[@ThanhTran-wf2jf]

This is one of the best tutorials i've seen in youtube videos. Very well-explained, very useful. Thank you for uploading this!

[@JohnnyChivers]

Thanks for the comment and supporting the channel.

[@JulioSerratos]

This tutorial is pure gold. Thanks

[@lyndsaymizen1881]

Great intro to AWS Glue for someone who's never seen it before. Needed to know these basics as we're moving to Glue for our data transformation/warehousing and this video does just that! Thanks!

[@JohnnyChivers]

Thanks for watching Lyndsay!

[@Ajay-nc9yn]

I have spent many hours on Udemy for Glue Job and Glue data Catalog, but after watching your video. I must say Damn Good Stuff, Sir!

[@engenheironomade]

Excelent tutorial. I made three others courses before that and only with this that I understead complete this tool

[@guilhermejf8642]

I just can't believe it's free here on KZbin! Great content, very well explained. Thanks!

[@JohnnyChivers]

Thanks for watching the video and a supporting the channel!

[@DataCraftBackbone]

As a skilled Ops Engineer / Developer i have search for infomartion about whats Glue and how its working in the basic and thanks so much for your video, its late me understanding the basic about what Glue are and whats it can be used for, :)

[@femaledeer]

Glad the tutorial took me through the console instead of running code.

[@1209youandme]

Superb course, covering all features with examples

[@JohnnyChivers]

Thanks for comment.

[@jairo_55]

I am using this in 2023, the interface has changed, but this still working, thank you so much for the video.

[@asifalimir3043]

Such an amazing video. Gives end to end idea about Glue in just 45 mins. Keep it up.

[@victors9585]

Johnny, you ROCK, man!!! 🔥👏🔥👏🔥👏🔥 Thank you so much and please don't stop spreading the wisdom of the GURU!!!! 🙏🙏🙏👍👍👏👏🚀🌟

[@hankbirkdale2154]

One of the best tutorials I've ever seen - let alone on a topic that is not easy.

[@AbhirupSenguptaUK]

your commitment shows in the quality!

[@diegogiardini]

It was very usefull for someone who is starting with AWS from scratch! Thank you :)

[@shelleycurrie764]

Johnny Chivers you rock. Really great intro to Glue.

[@Pavan-kn5pg]

One of THE BEST videos on AWS Glue. Thank you Johnny :)

[@dbeckett]

Finally an aws tutorial that uses "wee" and talks in a NI accent, made me laugh that I finally found a fellow NI person :) followed every word :D

[@GauravSakhuja]

Great tutorial in 40 mins you covered everything essential in Glue 👏

[@Rk23able]

Wonderful and neatly described in such a short video

[@venkatrao7868]

You are amazing and a natural teacher !!

[@javiermadriz7834]

Great video for beginners I hope to build some projects for keep learning

[@mohamedyasser5285]

Fantastic video, going to binge-watch your videos for the next few months!

[@JohnnyChivers]

Thanks for watching.

[@Josh-og9eo]

So excited to find this channel. Looking forward to watching all your videos!

[@ryoutausami6817]

Thank you. This is an awesome introduction to AWS Glue for beginners.

[@hemantmattoo]

Very nice and easy to learn video especially for begineers. Thanks for posting this.

[@sreddy5845]

Fantastic video: I have two questions 1. Can you create a partition key when importing data with a crawler? 2. The UI seems to have changed. The 'ETL job' has changed. Can you publish a refresher on that part?

[@jeevangangavarapu8683]

Excellent work Johnny. Keep continuing the work .

[@JohnnyChivers]

Thanks, will do!

[@akashdasgupta5088]

This is one of the best AWS Glue tutorial that I've come across on KZbin. Totally worth a like and sub❤️

[@JohnnyChivers]

Thanks for the sub!

[@ibinabobob-manuel323]

Towards the end of the course, I lost my way after we created the parquet file because my version is different from yours and I couldn't just get the visuals on my dashboard. Thank you so much and I will give this a try again some other time.

[@shadabbigdel5017]

Thank you very much for you great explanation and hands-on! It was very useful for me.

[@patriciacafundo1626]

Very good course, gave me a good view of glue AWS service

[@danielmdubois]

Do you have any resources or can direct me to any resources that can dig deeper into the topic skipped over @8:50, i.e., what someone would actually do with regards to IAM roles best practices?

[@ToddCunningham]

right when i needed this, always amazing content

[@JohnnyChivers]

Thanks for watching Todd

[@davidlean8674]

Great & comprehensive coverage of AWS Glue. I was hoping it did much more. Makes you appreciate how much easier, faster & cheaper it is to develop in Azure DataFactory. Where a single pipeline can consume multiple different CSV files, parse the filename, figure out the metadata on load, & redirect the output based on lookup parameters. All graphical or generated via script.

[@DataEngUncomplicated]

Hi David, This video did not cover AWS Glue Studio which provides more of a compressive graphical interface that I think would be more comparable to Azure Data Factory.

[@debarshiacharya3729]

This was a really nice video. Lot of learning within a very short time. Thanks

[@martinvuong6652]

Such an amazing tutorial! Thanks for getting this together. Looking forward to more content.

[@vjsnapp8178]

Amazing Content and style of teaching, Thanks a lot for making these videos.

[@vrlchebolu]

Great intro on AWS Glue! Thanks!!

[@gonzalea35]

This was awesome explanation of glue, thanks.

[@guilhermmontealto2160]

huge fan from brasil, thank you for your great content, it helped me a loooot

[@ambhat3953]

Very helpful for my studies

[@bitashamsss]

fantastic video, very helpful and useful. Looking forward to seeing more. keep up the great work and thank you so much.

[@AviinashP]

excellent video on glue

[@hazemzamalkawy14]

Thank you very much, I learnt a lot from this tutorial.

[@ammar9700]

Nicely done and explained. I have done my first AWS Glue job, without any issue... thumbs up bro 👍

[@JohnnyChivers]

Fantastic news Ammar!!

[@sriravi815]

Thanks for nice video, it helped me understand glue and setting up the pipeline in aws.

[@KoEDeath]

Impressive free content. Thank you!

[@JohnnyChivers]

Thanks for watching

[@maxmax2746]

Thank you for your work! Could you please help few little questions: 1) i saw, that while you it said that "customer_csv" table is partitioned, but "customer_parquet" is not (there is not "partitioned" mark nearby). So how should i make that files partitioned? Because it seemed, that you did the same thing for csv ant parquet, but get different result. Thank you in advance! 2) What would be happened, when you add another csv data for different day? how job will work? i didn't get, how jobs determine, that previous day already transformed to parquet, and newest day - no. And what if . As i understand - in your particular example from that video, that Glue job will be transforming from csv to parquet all the fyles inside customer_csv folder. But how to make it more determined based on run date? for example: in general i want to transform only previous loaddate files. As i understand, it should be done only in code of job

[@gabesusman4592]

this channels a great resource for learning data engineering on aws, it's been a big help. Keep up the good work Johnny!

[@roarmarketingconceptsllc5074]

This was a great tutorial! I really learned a lot about AWS Glue and I plan to leverage this knowledge to be more effective in my job tasks. Thanks so much, Johnny!!

[@obedrajugantala3489]

Good tutorial. I was able to follow and execute. Thanks Johnny!

[@maryo1134]

Amazing teacher..Many thanks for this

[@JohnnyChivers]

Thanks for watching.

[@PURAMSAIPRIYA]

Thank you for this video, it is very helpful gives a clear glimpse of AWS glue. This is a very important video for me for the interviews :)

[@kjewelson]

happy new year

[@JasonZhang-se2jo]

You are definitely a hero

[@JohnnyChivers]

Thanks for watching

[@Matrix_Mayhem]

Thankyou so much. Very informative Johnny!

[@anarossetto2742]

wonderful tutorial, great explanation, thank you! Unfortunatly aws glue job console has changed a lot and I could not finished the tuto :(

[@machinimaaquinix3178]

Thank you and well done. Was able to do the course still in June of 2023, though the ETL chapter was a bit of a challenge as AWS has completely redone those screens.

[@JohnnyChivers]

Great to hear!

[@michealdmouse]

How did you get past the ETL job chapter? The interface is totally different.

[@AkashBhosale-mr7kk]

Great Video great efforts. Thanks a lot for detail explanation. When I ran the crawler for 1st time as per video, it did not create partition column, I again created a new crawler with same details using same s3 folders and now it created a partition. What might the possible reason that it failed to detect at 1st time? Any key point to remember during building or some miss that lands us into such situation?

[@jriosfer]

It's very interesting your approach, I have a question, why does the glue's crawler from converts csv file to parquet format could not create the parquets table with a partition definition, which came from csv file?

[@JohnnyChivers]

Hi Jorge. The crawler will only create partitions when there is a folder. So something like s3://table_name/parition_1/file.csv will result in partition being created. However, crawlers can be a bit temperamental. They are some sort of ML algorithm and do go AWOL at times. For that reason I usually just create the tables through code manually - terraform, cloud formation or even DDL via Athena.. when it comes to real life use cases.

[@sagnikmukherjee5108]

Thoroughly enjoyed, Thanks.

[@ishabisht3685]

Thank you very much, it was very helpful video to learn end to end AWS Glue. 🙂

[@qadiralidanish7529]

Awesome tutorial.

[@daviluancarneiro6901]

Jesus bless you, my friend. Your explanations are very clear. My desire is that you continue making the difference, creating and publishing awesome contents that help people. Congratulations! Good job!

[@PatrickMcDonoughVanWash]

Highly recommend!

[@balledachandrahas8326]

Super helpful.... Thank you so much.

[@JohnnyChivers]

Thanks for watching!

[@benitinmagnate4937]

@25:00 "I'll talk about connections quickly", LOL! That's what AWS Glue, Azure Data Factory, SSIS, Informatica, are all about: CONNECTIONS! You are moving data from a source to a target, and to do that, you need to be connected to both, the source and the target. Basically, you are an S3 guy, LOL!

[@YogeshSeemakurthi]

Thank you for the course!! I want to create a custom classifier for text files is that possible?

[@JohnnyChivers]

Yes that possible. You’ll find it on the glue console under classifiers and ‘create’. I think there are some limitations on how customisable classifiers can be.

[@nishaddheeraj2]

really amazing and fun learning with Johnny :)

[@niranjanjamkhande3773]

Thanks for the video. We need videos for real world problems with real world data such as nested json. Can you guide on that?

[@JohnnyChivers]

Niranjan - something I can totally do and feel free to suggest ideas. For now there is an AWS Glue Library called Rationalize that can de-nest for you - aws.amazon.com/blogs/big-data/simplify-querying-nested-json-with-the-aws-glue-relationalize-transform/ There is also a pandas library that does exactly the same thing called json_normalize() and this available in glue out of the box - but don't use pandas in glue on bigger JSON loads.

[@muadddib4734]

Excellent content! Hope you keep it coming. I just subbed

[@JohnnyChivers]

Thanks for watching.

[@lukacindric8065]

What would be the video on your channel where you explain loading the data from s3 to dwh using Glue?

[@tullez01]

Amigo, muito obrigado pelo vídeo... Ficou ótimo! Abraços! Dear friend, thanks for this video. It's really great, helped a lot... Hugs from Brazil :)

[@JeffLentz]

Fantastic! Thank you for putting this together. It helped me a lot.

[@deetechbee]

awesome video!

[@claudioruz]

Good training course, thank for all

[@JohnnyChivers]

Thanks for watching.

[@HughMcBrideDonegalFlyer]

Great video Johnny , my only complaint would be video resolution , It can be a bit of a strain to try to watch this on a laptop

[@TheSimpGatsby]

thanks mate, u made my day.

[@BEBKomalTeke]

Hi, there's one option in ETL as jobs and jobs legacy. Could you explain what's the difference?

[@JohnnyChivers]

The “jobs” tab is new where AWS look to be consolidating all functionality to do with jobs. The “legacy” tab is where jobs have been located for the last 4+ years. It looks like most of the legacy functionality has been migrated to the jobs tab - albeit the UI is slightly different. This change only happened about 2/3 months ago and is all pretty new. I suspect the legacy tab will be removed eventually, but only once all functionality has been migrated to the new ‘jobs’ section.

[@Straight-Data-Science]

Very well done! Thanks for sharing this!!

[@augustogoldner4267]

That was fantastic, mate! Thank you very much for sharing your knowledge!