Mastering AWS Glue Unit Testing for PySpark Jobs with Pytest

[@thepravinbtech]

Hi Data eng your knowledge in AWS and way of teaching is excellent could you please share the videos on CICD pipeline to deploy the glue jobs to production

[@DataEngUncomplicated]

Thanks for the kind words! Yes actually this was going to be one of my next videos. How to deploy a glue job to terraform with terraform.

[@0777deep]

Thanks !

[@Angleito]

how do you add third party python libraries ?

[@DataEngUncomplicated]

I don't know an elegant way to do this but you can go into the docker container and install the python libraries you need directly that way.

[@harshadk4264]

Do you use the Factory Design pattern?

[@renyang2320]

Your functions based job is quite straightforward. Would you like to organize your glue job in a Python class?

[@DataEngUncomplicated]

I made the script just for this KZbin video, sure things could be organized into classes if it makes sense?

[@kckc1289]

How would you recommend local dev and organization -> uploading to AWS for scripts with multiple files ?

[@kckc1289]

Do you have a Github for this Pytest example?

[@DataEngUncomplicated]

Hey, checkout my videos on local development for AWS glue. I covered topics like using interactive sessions, pycharm and vs code with a docker container with AWS glue. In order to upload them, I recommend managing them with IaC with terraform or cdk.

[@joseluisvega3237]

I've been looking to develop some unit tests with pytest but I would like to mock everything related to the Glue Environment. I've been trying to do it through MonkeyPatch but the problem I have is when I transform the dybamicframe to dataFrame, it's also expecting a full mock of the dataFrame and it's functions. Any experience with that?

[@DataEngUncomplicated]

Hi, Can you explain how your approach is different than how I created the unit test in the video? If you design your functions to do one particular thing, it makes it much easier to write unit tests for it.

[@joseluisvega3237]

The approach is to be able to run the unit test without a glue environment, no docker image, pure local développement (my laptop). Mocking GlueContext and DybamicFrame. The tests would use the mocks of these instances so there's no interaction with AWS glue at all.

[@DataEngUncomplicated]

Yea I don't know how you can achieve this. your environment you are running the glue jobs need to have the python libraries installed so you can execute the code. The way I set it up is I am doing 100% local development but glue is in a docker container. If you can't use docker, you need to install and set up spark directly on your local machine. I tried to do this following the documentation but it was messy and I couldn't get it to work in the end