AWS S3 File Upload + Lambda Trigger - Step by Step Tutorial

Learn how to invoke a Lambda function in response to a S3 File Upload in this Step by Step Video tutorial.
Looking to get hands on experience building on AWS with a REAL project? Check out my course - The AWS Learning Accelerator! courses.beabetterdev.com/cour...
🎉SUPPORT BE A BETTER DEV🎉
Become a Patron: / beabetterdev
📚 MY RECOMMENDED READING LIST FOR SOFTWARE DEVELOPERS📚
Clean Code - amzn.to/37T7xdP
Clean Architecture - amzn.to/3sCEGCe
Head First Design Patterns - amzn.to/37WXAMy
Domain Driven Design - amzn.to/3aWSW2W
Code Complete - amzn.to/3ksQDrB
The Pragmatic Programmer - amzn.to/3uH4kaQ
Algorithms - amzn.to/3syvyP5
Working Effectively with Legacy Code - amzn.to/3kvMza7
Refactoring - amzn.to/3r6FQ8U
🎙 MY RECORDING EQUIPMENT 🎙
Shure SM58 Microphone - amzn.to/3r5Hrf9
Behringer UM2 Audio Interface - amzn.to/2MuEllM
XLR Cable - amzn.to/3uGyZFx
Acoustic Sound Absorbing Foam Panels - amzn.to/3ktIrY6
Desk Microphone Mount - amzn.to/3qXMVIO
Logitech C920s Webcam - amzn.to/303zGu9
Fujilm XS10 Camera - amzn.to/3uGa30E
Fujifilm XF 35mm F2 Lens - amzn.to/3rentPe
Neewer 2 Piece Studio Lights - amzn.to/3uyoa8p
💻 MY DESKTOP EQUIPMENT 💻
Dell 34 inch Ultrawide Monitor - amzn.to/2NJwph6
Autonomous ErgoChair 2 - bit.ly/2YzomEm
Autonomous SmartDesk 2 Standing Desk - bit.ly/2YzomEm
MX Master 3 Productivity Mouse - amzn.to/3aYwKVZ
Das Keyboard Prime 13 MX Brown Mechanical- amzn.to/3uH6VBF
Veikk A15 Drawing Tablet - amzn.to/3uBRWsN
🌎 Find me here:
Twitter - / beabetterdevv
Instagram - / beabetterdevv
Patreon - Donations help fund additional content - / beabetterdev
#AWS
#S3
#Lambda

[@justsomeguy1408]

I appreciate the step-by-step walkthrough. I was just wondering how to do this for a project I'm working on and this makes it really easy to understand and follow along. Thank you!

[@BeABetterDev]

You're very welcome!

[@abhijeetpatil9883]

Super simple and really helpful. Thank a ton!

[@qiaoxuanhu1447]

thank you so much, it's extremely helpful with what i am working on.

[@darimco3660]

Very cool explanation. Thank you.

[@samk_jg]

very nice tutorial!

[@johnapple3471]

You deserve a diamond play button. Thank bro!

[@BeABetterDev]

One day :) Thanks!

[@Rafaelkenjinagao]

Thank you! 👏

[@lnkrishnan]

Any chance you can do a deep-dive into the core concepts of S3 - such as how encryption at rest works under the covers, what happens when we upload files encrypted with KMS keys, how does S3 encrypt the content, versus when we just use sse-s3 encryption, etc.

[@sakirbey7911]

thanks for the video.

[@BeABetterDev]

You're very welcome!

[@gshan994]

I had done s same thing using S3 recent motifs in cdk

[@abdelhadidjafer7004]

your videos are gold , but we need a node version of it , that would be super easy for you to do but very helpful for us beginners

[@taocai5997]

Thanks! Very good tutorial. I have an issue with the POST, Put is fine since I upload it from the console. Also, "csv[1]" gives an error.

[@ankitanurag]

If the bucket has some elements already, error is thrown while creating the trigger, but if the bucket is empty, trigger is created successfully.

[@mayanktripathi4u]

Thanks for this video, it really helps. Please do let know how to pass additional data to event object when triggering from S3. Like apart from bucket name and file name (which are default) I need to pass some additional info as well.. how to do this. Please help. something similar to event object we modify w.r.t AWS API Gateway.

[@AliHasan-bw7km]

Code link?

[@ruggeropiazza2157]

Great video! But I'm keep getting an Access Denied error when calling the GetObject operation. I added the permission to the role as shown in the video.

[@on_my_way_to_ML]

You are working on a Serverless application where you want to process objects uploaded to an S3 bucket. You have configured S3 Events on your S3 bucket to invoke a Lambda function every time an object has been uploaded. You want to ensure that events that can't be processed are sent to a Dead Letter Queue (DLQ) for further processing. Which AWS service should you use to set up the DLQ? what about this question answer？

[@oseiasibeybaffour-awuah3770]

Hi, thanks for detailed video. What if I want lambda handler to call methods on external class and load new file/object into memory. For instance. class VersionClass( Amazon s3Client, int version) { public loadNewObject() { /// uses s3Client to load new object into memory } } class VersionClient( VersionClass versionClass ) class Handler implements Requesthandler { public T handleRequest(T, context) { versionClass.loadNewObject(); //posible for Handler to know this versionClass object } } In short, is it possible to set lambdas so they can update initialized external classes?

[@SafathN]

Can I parse event to extract absolute file path, including bucket name, subfolder name that I specified on prefix section ? how do I get the absolute path of the file from event ?

[@pramodbasu7732]

I am getting access denied on the s3 bucket when lambda is triggering the s3

[@user-pz8oc4vs9s]

Where can I find policies you used in the video?

[@856shaileshkumar]

Hi bro , can you please post the code link . Writing on our own has a greater chance of mistake

[@Obemfal]

what was the role of the lambda trigger? why did you have to upload the .csv file manually? is there a way to trigger a file upload to s3 without human intervention?

[@HeavyDriver8214]

Hey, I am using the same code line #11 & #12 to get the bucket & file name, but it throws key error “Records” doesn’t exist in the event dictionary. Any idea what could be the problem?

[@zarifkhan5788]

where can I get the link to the code ?

[@shubhammahure3530]

Fix the issue of job failure when all the market zip files are placed in cft folder(s3 bucket) Some of the jobs will fail due to concurrency issue Exceeded maximum concurrent capacity for your account:500. How i add Queue for this jobs and delay so that it cannot exceed 500 DPUs.How check how many files are running.

[@user-ml9tu8tm1o]

Please share git hub link to access the code used in the video

[@eamonkelly215]

Is there a link to your GitHub? Thanks

[@BeABetterDev]

Hi Eamon, here's the link: github.com/beabetterdevv Cheers

[@zieddatascientist546]

there are no import boto3, you have to had an error, but it seems be cutted !

[@manu93ize]

i am still confuse about event part.

[@laythabdulkareem1887]

dude where the python code and csv file? where can I find them???

[@rrchaitanya9559]

where is the code link

[@pradeeppk-ns1so]

Hi I'm getting this error

[@hibamechyakha3526]

I followed this step by step and I still get the error : "An error occurred when creating the trigger: Unable to validate the following destination configurations" when trying to create the trigger S3

[@manu93ize]

check bucket properties and detach lambda finction

[@Abhishekkumar-yn5td]

Log group does not exist The specific log group: /aws/lambda/file-upload-trigger does not exist in this account or region. I am getting this error. Can anyone please help rectify the mistake?

[@StevenCrickman92]

I was getting this error because the lambda function was not being invoked when I uploaded a file. I changed the event type from "POST" to "All object create events" and the lambda function was invoked and the logs started coming in. I think this is because the upload event doesn't necessarily use a POST call.

[@sjc3hw106]

Thank you for the video. I followed your video, but I am unable to see any log from the print() statements. I was able to run the "Test" and see the log group and a log entry, and see the code worked fine. However, when I uploaded a file, there was no log. I tried manual upload from s3 console, use aws cli, and use boto3. None seemed to produce any log. I have watched your video multiple times and don't think I missed any step. Also before I did any "Test", I had a red message "Log group does not exist". Looks like Ari. G also had the same issue in the related older video kzbin.info/www/bejne/bZrJhqx6pJiDmqs. Is there some setup that we should be aware of?

[@luciofondon1104]

I'm having the same issue

[@BeABetterDev]

Hi there, I ran into a similar issue during testing. The way I resolved it was going in to the S3 console and finding the events section. You should see an entry that indicates your bucket is connected to your lambda function for POST events. Remove the connection, and then go and re-create your Lambda function and add your S3 bucket as the trigger. Hope this helps.

[@sjc3hw106]

@@BeABetterDev Thank you very much. This fixed it.

[@pradeeppk-ns1so]

specific log group does not exist

[@StevenCrickman92]

I was getting this error because the lambda function was not being invoked when I uploaded a file. I changed the event type from "POST" to "All object create events" and the lambda function was invoked and the logs started coming in. I think this is because the upload event doesn't necessarily use a POST call.