Develop AWS Glue Jobs Locally Using Visual Studio Code and Docker on Windows - step by step

[@ParagJadhav0]

Now this is what you call a STEP BY STEP tutorial. Kudos to you sir 🍻. Please accept my humble subscription

[@DataEngUncomplicated]

Haha Thanks for subscribing!

[@ParagJadhav0]

@@DataEngUncomplicated Can I have more than one script for the same job?

[@DataEngUncomplicated]

​@ParagJadhav0 absolutely! It would be a deal breaker if we couldn't call additional classes for functions from other files. You just need to import the scripts or functions in your glue job. I do it all the time.

[@bananaboydan3642]

Your channel is amazing bro. Perfect for the project I’m working on

[@DataEngUncomplicated]

Thanks a lot!

[@prabhathkota107]

Beautifully explained about the setup. Understood how docker works as well. thanks a ton, Subscribed.

[@tapasdalai5115]

Thanks its a great effort given by you. Good work keep it up

[@shashankreddy8390]

Hi, I love all your videos. I have a video suggestion: Can you please make an end to end project on 1.how to run a job from start to end. 2. How to monitor glue cloud watch metrics? 3. How to check cloud watch error logs if a glue job has failed or is a success? 4. How to trigger this glue job daily and append the results into an s3 bucket.

[@PuzzlerCraft]

Thanks a lot to put out a comment in the previous video.

[@HenriqueCalasans]

Nicee! thank you so much!

[@DataEngUncomplicated]

You're welcome!

[@dinhhoangtu311]

Well explained video. Thanks!

[@DataEngUncomplicated]

You're welcome!

[@SonPhan1]

i appreciate the really informative video! I followed everything and i'm stuck on running the pyspark code in the dev container environment. when i launch the dev container in the same/new window, i don't see the extensions in the container environment. The python interpreter doesn't show up either and when i go to the extension tab in the container environment, all the extensions are not installed. Is there additional configuration files in vs code i need to modify to enable the already installed extensions to run from the dev container?

[@julianromero3359]

Excelent tutorial

[@DataEngUncomplicated]

Thanks Julian!

[@DevNarayan51]

I am using MAC OS and trying to setup AWS Glue locally with VS code. Can you help me please. As i am not able understand to setup in my mac after 3:22

[@DataEngUncomplicated]

For this step, just navigate to where you have your vs code workspace file and open it in vs code. It might look differently on a Mac I did this on a windows machine.

[@rohithreddy41]

thank you for the video. I am unable to run the program because i do not see the run button after clicking "attach in current window"

[@rohithreddy41]

I had to install python in the container and now i see the run button.

[@keyurpatel3387]

Thanks. However, I am getting the below error while following this in Windows 11. Any idea what could be wrong and how to solve it: def since(version: Union[str, float]) -> Callable[[F], F]: saying invalid syntax

[@DataEngUncomplicated]

Strange, at what stage did you get this error?

[@watsup1269]

Hi, to someone who found this error. I also did but solved it by using the command: "python3 " instead of "python "

[@OmegaStyler]

Great work Adriano, thanks for these videos! Quick question about AWS Glue - lets say I have 300 tables in my Data catalog and I want to load them all at once, add a loaddate column to all of them and write them into another target (Another Database / data catalog etc.) What is the easiest solution to do this? I'm able to load one single table but not all at once. Many thanks!

[@DataEngUncomplicated]

Thanks, if your doing the exact same transformation on all 300 tables, I would put the table names in a list and then loop through each table to add this load date column and then write it to another target. If your tables are very large I would be careful trying to bring all 300 tables into memory at once so perhaps looping through might be a better option of this is the case. The other option is Create a Parameterized Glue Job...Write a Glue ETL script that performs the required transformation on a single table, making use of parameters for dynamic values (e.g., input table name, output table name, etc.).

[@juandavidpenaranda6136]

@@DataEngUncomplicated amazing answer, thanks

[@TheSmilingLamp]

How do you fix Permission Denied issues? I can't mount my .aws folder or any other directory to the workspace and also be able to read/write files to these mounts

[@DataEngUncomplicated]

Hi, it sounds like you don't have admin permissions on your own machine. I would probably talk to your IT team to see how they could help. I didn't experience any permission issues.

[@Neeoooo]

@@DataEngUncomplicated It looks like the glue_user does not have permission to access the mapped volume folder location for AWS configs. I had to specifically give read permissions from my local machine

[@calendr13]

Great Video! But I cannot run the python script, the play button does not appear

[@rahulpanda9256]

This video is not clear.. especially the workspace section. Can you try to create a new workspace and show the same? Also how would developers use a common code base in this case as the workspace location is internal to glue. How can we mount an s3 bucket to save everyone’s code?

[@DataEngUncomplicated]

Hi Rahul, The workspace location is specific to the individual developer. When we run code on our docker container, we are telling the docker container to run some code we have on our local machine. So to answer your question about "how would developers use a common code base". This should always be in a code repository like github, codecommit, or bitbucket. I wouldn't advise an s3 bucket for storing everyone's code.

[@stevenjosephceniza8245]

Thank you for this guide! I tried using pycharm and my old computer cannot handle it. I almost purchased for a subscription.

[@DataEngUncomplicated]

Hi, I'm not sure what you mean it almost purchased a subscription. But you need pro to use docker in pycharm.

[@NitaMote-g1j]

How run glue job by passing job argument?

[@DataEngUncomplicated]

It depends how are you planning to trigger your glue job. Do you have an idea?

[@RyanMontSerrat]

Great video man, but how do I import local csv files?

[@DataEngUncomplicated]

Hi Ryan, since your deployed AWS glue eventually runs in the cloud you would need your data in S3. You can manually upload your data to S3 through the AWS S3 console or use boto3 or the AWS S3 cli to do this.

[@ashishrakhit5799]

Can you please, share the link to your github repo? Thanks

[@DataEngUncomplicated]

github.com/AdrianoNicolucci/dataenguncomplicated

[@Levy957]

you are awesome

[@jzevakin]

Thank you!!

[@DataEngUncomplicated]

You're welcome!

[@dipeshg8581]

Thanks for the video. But sir we are not able to display dataframe as table . It's showing json line by line. Sir How to resolve this ?

[@DataEngUncomplicated]

That's a good question, I'm not sure how to visualize the data in vs code from spark I use pycharm mostly.

[@dipeshg8581]

@@DataEngUncomplicated sir on pycharm also dynamicframe is shown as json line-by-line in your video

[@dipeshg8581]

@@DataEngUncomplicated Sir On pycharm also dynamic-frame is not shown as table instead it is json line-by-line as seen in your video.

[@DataEngUncomplicated]

I don't think pycharm supports visualizing spark dataframes, you can bring the data into a pandas dataframe and then you can view the data in debugging mode as one method. @@dipeshg8581

[@dipeshg8581]

@@DataEngUncomplicated thank you sir 🙏🏼

[@maximilianrausch5193]

What is different about this new video compared to the prior ones?

[@DataEngUncomplicated]

This one is with visual studio code vs my last video was pycharm. I got a lot of comments that more people use vs code and wanted a tutorial specifically for that ide

[@Angleito]

does this work with the debugger?

[@DataEngUncomplicated]

Yes