How to Install Pandas on AWS Lambda Function

[@DrNickBailey]

Oh my. After, I can't tell you how many medium posts, stack overflows, and rubbish AWS documentation I've trawled through, nothing was successful except this. THANK YOU.

[@DataEngUncomplicated]

Haha you're welcome nick! I'm glad you got it working.

[@DrNickBailey]

@@DataEngUncomplicated I've already linked this video on two stack overflows. Seriously, I can't understand why the AWS documentation is so appallingly crap and gave no idea as to the steps you gave here. Yours is such a short but perfectly put together video.

[@brunocarvalho3229]

You're a life savior my guy, thank so much for this. I struggled for days trying a bunch of different solutions without any success until i got here. Happy 2022 btw

[@DataEngUncomplicated]

Happy 2022 Bruno! I'm glad you got it working! Thanks for the feedback 😊

[@DaveFanch1022]

Holy shit, I spent days trying to figure out how to get “numpy” installed. Awesome video !

[@DataEngUncomplicated]

Thanks Dave!

[@naveshkumar]

the only you tuber who makes sense. kudos and great success be upon you!

[@DataEngUncomplicated]

Haha thanks Navesh, I'm glad my video was helpful

[@dinesh_vijay]

Thank you for the step-by-step instructions

[@gotchaxel]

Thanks for the explanation. Unfortunately this does not work for me, still getting "Unable to import module 'XXXXX/lambda_function': No module named 'pandas'" error.

[@DataEngUncomplicated]

Did you attach the lambda layer to your python function? You might have missed a step

[@gotchaxel]

@ yes I did. I will provide a summary tomorrow but I checked every step very carefully.

[@karolinedutra4276]

I'm getting the same error, did you find a way to solve, @gotchaxel?

[@gotchaxel]

@ no, it drove me insane so after wasting 6hrs on it, reading aws ressources and tutorials to no avail, I gave up. And working with tables without Panda sucks haha!

[@karolinedutra4276]

@@gotchaxel Yes, it really sucks! hahaha I gave up to do some thing similar to the video, but I found another way, maybe it can be useful for you. I changed the runtime of my function from Python 3.13 to Python 3.9, thus in the AWS layers options, as he showed in the video to Scipy, There is one called "AWSSDKPandas-Python39" which has Pandas, naturally. So I chose this one and pandas worked. Well, if the Python version isn't an issue for you, maybe you can try this. :)

[@FlavorOfTheMonthChannel]

For the error: 'wheel' is not recognized as an internal or external command, I fixed mine by running "pip install --upgrade wheel". I also changed my command prompt to point to my python scripts directory.

[@DataEngUncomplicated]

Thanks for sharing!

[@FlavorOfTheMonthChannel]

@@DataEngUncomplicated Thanks for your video! It saved me a ton of time!

[@amirhussain549]

Thank you for great tutorials, after two day struggling I found you video and solve my problem

[@DataEngUncomplicated]

Thank you Amir! I'm glad you got pandas installed successfully!

[@shrinidhig3672]

OMG... ur awesome... i just followed u step by step and it resolved my issue.. thanks a lot dude 😊

[@DataEngUncomplicated]

Haha thanks! You're welcome Shrinidhi! Glad you got it working!

[@ProphecyMzwah]

Your videos are great. Thank you very much.

[@DataEngUncomplicated]

Thanks for the kind words

[@SachinKumar-y4e1b]

Thanks buddy,It helps me alot. It covers everything I need.

[@DataEngUncomplicated]

You're welcome!

[@AbhinavBhatnagar1]

Informative videos… 👌🏻… looking forward for more

[@DataEngUncomplicated]

Thank you Abhinav!

[@nlopedebarrios]

Now AWS includes AWSSDKPandas-Python312 so it's easier to add pandas to your Lambda function. However, I'm getting "Missing optional dependency 'fsspec'. Use pip or conda to install fsspec." I've follow these steps to install the latest version, but failed: "Unable to import module 'lambda_function': No module named 'fsspec'". Any suggestions?

[@ChinmayaPradhan-id1qs]

wheel is not recognized as internal external cmd error

[@FlavorOfTheMonthChannel]

I fixed mine by running "pip install --upgrade wheel". I also changed my command prompt to point to my python scripts directory.

[@ahmedsaied8373]

Thanks for sharing, but I can't find AWSLambda-Python37-SciPy1x Layer in down list

[@DataEngUncomplicated]

Hi Ahmed, make sure you are not searching in "custom layers" This is a layer that is provided by AWS.

[@sudipbala9647]

I had the same situation. It was because the lambda fun Runtime was Python 3.9 .. Once you change your Runtime to Python 3.7, the option will be available on dropdown.

[@NextLevelMicrosoft]

For those who want to stay at Python 3.9, is there another whl file required besides the one for numpy to make this work? like how we needed to grab the pandas whl file and the pytz whl file?

[@MarynaLongnickel]

I'm getting an error "errorMessage": "Unable to import module 'lambda_function': No module named 'pandas._libs.interval'",

[@mengcenli9890]

Useful instruction👍 It assisted me with running lambda python codes.

[@DataEngUncomplicated]

Thanks for the comment Mengcen!

[@martinvuong6652]

Awesome content! Thanks. Followed same instructions to get ndjson package layer in.

[@DataEngUncomplicated]

Thanks Martin! I'm glad it put you on a path for another package to work!

[@martinvuong6652]

@@DataEngUncomplicated Hi Adriano, referenced this guide again today for another library and it worked like a charm! One question, what is the purpose of pytz library? Is it necessary to always include it in our custom layer package? Thank you :)

[@laiyawang2890]

hi i am using python 3.13, when i am at the last step , there is no aws Lambda layer of python showing on the list, should i down grade my running python version ?

[@DataEngUncomplicated]

Are you saying that they haven't added the scifi layer to AWS yet? Downgrading should work. They must have added 3.13 very recently

[@anuragsingh711]

why i am getting error saying unable to import required dependencies : numpy : module error named : numpy

[@DataEngUncomplicated]

You missed the step in the video to handle that issue that comes up.

[@anuragsingh711]

@@DataEngUncomplicated thanks you so much pointing that out

[@shantanutiwari686]

I followed the same process but getting the error : Unable to import module 'lambda_function': No module named 'pandas'

[@DataEngUncomplicated]

Hi, it should work. It's possible you downloaded the wrong file?

[@tejaswi3046]

I am still facing with the numpy import error and even used AWSLambda-Python38-SciPy1x library , but unable to resolve , kindly let me know if any inputs

[@DataEngUncomplicated]

Strange, sorry it worked for me and others from the lambda layer I selected. Try selecting the specific version I had perhaps?

[@saptanilchowdhury1851]

I am not able to unpack wheel file using wheel file it is telling me that wheel is not recognized as internal/external command

[@DataEngUncomplicated]

Hey, do you have the library installed to unpack the wheel file?

[@yaasirahmed6467]

Hello, I did the same thing but for scikit-learn. When I do the command in the lambda function from sklearn.model_selection import train_test_split it returns a module error, "No module called sklearn", any fixes?

[@DataEngUncomplicated]

Hi, I 'm not familiar with the installation process for scikit-learn. maybe check our their documentation page to see what you are missing?

[@indiajackson5959]

I got an "errorMessage": "Unable to import module 'lambda_function': No module named 'packaging'", I don't know what to do?

[@DataEngUncomplicated]

Hi India, Make sure the pandas wheel version matches the version you are using on lambda. For example, if you want to use python 3.7 on lambda, make sure to download the 3.7 wheel version of pandas. I'm not sure if this will solve this exact issue but worth a try.

[@davidrvu]

Thank you very much! This helped me a lot!

[@DataEngUncomplicated]

You're welcome David! I'm glad it was helpful.

[@guilhermegarcia1154]

No module named 'pandas._libs.interval' not built

[@DataEngUncomplicated]

Hi Guilherme, what version of pandas were you trying to install? I have not seen this error before.

[@mengcenli9890]

I encountered the same issue. However, I solved it by choosing the proper python version. For example, if your lambda function use runtime Python 3.8, then you must use the right wheel package of cp38. Hope this would help.

[@DataEngUncomplicated]

@@mengcenli9890 Thanks for leaving this note, I'll make sure to pass this along if someone else has the same issue.

[@JasonZhang-se2jo]

Hi There , Thank you for your great video and training first. When I am running the "wheel unpack", it is returned the error of "access is denied" in my Windows Command line, could you help to shed some lights on this , thank you in advance

[@DataEngUncomplicated]

Hi Jason, yes, it looks like you don't have access to that location on your machine.

[@mensmans9241]

Fantastic help! Thank you so much!

[@DataEngUncomplicated]

You're welcome! Glad it was helpful.

[@patricechaula3430]

Thank you, you are a life saver.

[@DataEngUncomplicated]

You're welcome!

[@nishant86]

Thanks man it worked. :)

[@khushimourya397]

can you please tell if I have python version 3.9 in my windows so I have to choose python 3.9 runtime in AWS lambda?

[@DataEngUncomplicated]

Hi Khushi, There should be no relationship between your python version on your Windows machine and what version of python runtime you want to use in AWS Lambda. Make sure you download the correct Pandas version to match the lambda runtime you want to use.

[@khushimourya397]

@@DataEngUncomplicated ya I did it but when I uploaded zip file it failed to create the layer.

[@DataEngUncomplicated]

As mentioned in the video, make sure you are downloading the Linux distribution and not the windows one since lambda does not run on windows machines

[@khushimourya397]

@@DataEngUncomplicated Okk I will try with that again. Thanks.

[@divyadeivendran2219]

we have to download only wheel file? Then what about .tar.gz

[@DataEngUncomplicated]

Correct, we only need to download the wheel file and unpack it. You can ignore the .tar.gz file.

[@sso2245]

It didn't work for me... I followed your tutorial but it still shows error message "no module named pandas"

[@DataEngUncomplicated]

Hi sso, did you link the lambda layer to your python function? there are a few places where if a step was done incorrectly, it won't recognize pandas python library that was uploaded as a lambda layer.

[@sebastosaurio]

this is great, thank you very much!

[@DataEngUncomplicated]

You're welcome Sebastian.

[@parthvaghani702]

Worked for numpy issue BTW On this date I didn't had to create custom layer for pandas as well I attached AWSDataWrangler... inbuilt layer and it resolved issue of pandas and numpy import

[@DataEngUncomplicated]

Hi Parth, AWS has added an input layer for AWS Data Wrangler which is great but I don't think it's available for all python versions. Also If you need to lock in pandas at a specific version then you may still want to create a custom layer or you are going to be stuck with whatever version aws data wrangler is using.

[@lacoletenay]

Thanks it helped me a lot! One question, do you have a video how to import/install your own built python package to aws lambda? I mean if you have a file with setup.py and everything what is needed

[@DataEngUncomplicated]

Hi ladislav, glad this was helpful. Unfortunately I don't have a video on how to install your own built python package to aws lambda but this seems like a great idea for a future video. I'm going to add it to my future list.

[@vijendersinghshekhawat6244]

This was very helpful..

[@DataEngUncomplicated]

Thanks Vijendrr!

[@mahanteshangadi9487]

awesome ... great stuff

[@DataEngUncomplicated]

Thanks Muhantesh!

[@beehack]

thanks man love you 3000 i was able to upload custom package and sove my problem

[@DataEngUncomplicated]

You're welcome my friend! I'm glad it was helpful!

[@johnnysim1985]

THANK YOU!

[@DataEngUncomplicated]

You're welcome John!

[@henriquesglima]

Thanks so much. Just making sure people wont have issues as I did, but at 3:10, the folder name MUST be python otherwise it wont work.

[@DataEngUncomplicated]

You're welcome! Great call out.

[@Joey-mo1tv]

Thank you so much! Clear and easy easy to follow

[@DataEngUncomplicated]

Thanks Joe!

[@zouaouimahdia7562]

I HAVE THIS ERROR : Response { "errorMessage": "Unable to import module 'app': No module named 'pandas._libs.interval'", "errorType": "Runtime.ImportModuleError", "stackTrace": [] }

[@DataEngUncomplicated]

Hi Zouaoui, Make sure the pandas wheel version matches the version you are using on lambda. For example, if you want to use python 3.7 on lambda, make sure to download the 3.7 wheel version of pandas.

[@zouaouimahdia7562]

@@DataEngUncomplicated exactly i changed the version and it works thank's

[@bellie888]

very helpful :)

[@DataEngUncomplicated]

Thanks for the feedback Queanbeyan.

[@raviteja-op1nk]

insted of all these you could have placed the zip file link in the description

[@DataEngUncomplicated]

Hi Ravi, the zip file will become out of date as new versions of pandas is being released. Unfortunately I don't have the capacity to maintain all these versions

[@eugeniosp3]

Good god you were made to teach homie

[@DataEngUncomplicated]

Haha thanks, I appreciate the kind words!

[@abel_nix8341]

Thanks a lot!

[@DataEngUncomplicated]

You're welcome!

[@larie22]

FYI if anyone is watching this in 2024, you can just add the pandas layer without uploading anything. Skip to the end where he is adding the "numpy" layer. It's the same now for pandas.

[@DataEngUncomplicated]

Correct, the aws sdk for pandas layer comes with pandas. However, this tutorial is relevant if people can't use lambda layers for whatever reason or need a specific version of pandas they need to install.

[@pranay619]

@@DataEngUncomplicated @larie22 I want to use scikit learn package ,should I make custom layer for scikit learn and use along with aws provided layers for pandas and numpy?

[@aakashkag5239]

Thank you very much! This helped me a lot!

[@DataEngUncomplicated]

You're welcome. I'm glad it was helpful!

[@jonathangallant-mills6434]

Thank you!!!