AWS DynamoDB Streams to Lambda Tutorial in Python | Step by Step Guide

DynamoDB Streams are a powerful feature that allow applications to respond to change on your table's records. Combining this feature with Lambda functions creates a highly scalable and fault tolerant application that can easily respond to record modification. In this video, I show you how to create a Lambda function and define a handler function (in python) that will respond to DynamoDB change events. From there, I define a required IAM role and attach it to the Lambda Function. Afterwards, I create a DynamoDB table and hook up the Lambda trigger. I test the connectivity by making manual updates on my DynamoDB table and confirm the Lambda function is responding appropriately.
Looking to get hands on experience building on AWS with a REAL project? Check out my course - The AWS Learning Accelerator! courses.beabetterdev.com/cour...
📚 MY RECOMMENDED READING LIST FOR SOFTWARE DEVELOPERS📚
Clean Code - amzn.to/37T7xdP
Clean Architecture - amzn.to/3sCEGCe
Head First Design Patterns - amzn.to/37WXAMy
Domain Driver Design - amzn.to/3aWSW2W
Code Complete - amzn.to/3ksQDrB
The Pragmatic Programmer - amzn.to/3uH4kaQ
Algorithms - amzn.to/3syvyP5
Working Effectively with Legacy Code - amzn.to/3kvMza7
Refactoring - amzn.to/3r6FQ8U
🎙 MY RECORDING EQUIPMENT 🎙
Shure SM58 Microphone - amzn.to/3r5Hrf9
Behringer UM2 Audio Interface - amzn.to/2MuEllM
XLR Cable - amzn.to/3uGyZFx
Acoustic Sound Absorbing Foam Panels - amzn.to/3ktIrY6
Desk Microphone Mount - amzn.to/3qXMVIO
Logitech C920s Webcam - amzn.to/303zGu9
Fujilm XS10 Camera - amzn.to/3uGa30E
Fujifilm XF 35mm F2 Lens - amzn.to/3rentPe
Neewer 2 Piece Studio Lights - amzn.to/3uyoa8p
💻 MY DESKTOP EQUIPMENT 💻
Dell 34 inch Ultrawide Monitor - amzn.to/2NJwph6
Autonomous ErgoChair 2 - bit.ly/2YzomEm
Autonomous SmartDesk 2 Standing Desk - bit.ly/2YzomEm
MX Master 3 Productivity Mouse - amzn.to/3aYwKVZ
Das Keyboard Prime 13 MX Brown Mechanical- amzn.to/3uH6VBF
Veikk A15 Drawing Tablet - amzn.to/3uBRWsN
📚 References / Resources:
Getting started with AWS: • Introduction to AWS | ...
Lambda function code (python): gist.github.com/djg07/2bcf216...
☁Topics covered include:
- Lambda Function Creation
- Lambda handler using python
- Lambda DynamoDB Streams IAM role
- DynamoDB Table Setup
- DynamoDB Streams
- DynamoDB NewImage
- DynamoDB OldImage
- DynamoDB Stream "Change" Events
- Lambda Cloudwatch Monitoring
🌎 Find me here:
Twitter - / beabetterdevv
Instagram - / beabetterdevv
Patreon - Donations help fund additional content - / beabetterdev
#DynamoDB
#DynamoDBStreams
#Lambda
#AWS
#Serverless

[@malikahmed7045]

You are amazing and extremely intelligent, how come I never found your channel before. Why did you stop posting new videos? You could very easily be an inspiration my friend for millions of people out there seeking knowledge

[@anandakella5283]

This is one of the best video tutorial for integrating Dynamo Streams and Lambda. Thank you.

[@BeABetterDev]

Thank you so much for your kind words. It means a lot to me.

[@GuitarreroDaniel]

I love you man. This video was so easy to follow and I was able to made my own implementation at the first try. Thanks for everything!

[@mahmoudebada4025]

Thanks, A lot, Really it is the most helpful video I've found in youtube for AWS Lambda

[@BeABetterDev]

Mahmoud, thank you for such kind words. Your support keeps me motivated to make these videos!

[@lopper1904]

Thanks for the useful video! it's a gem in the forest of low quality tutorials! I've a topic suggestion: connecting Lambda to RDS, not only covering the code, but also the VPC challenge and thinking about request limitations and handling the risk of lambda functions choking non-serverless services like RDS. Thanks again!

[@BeABetterDev]

Hi Lieven, I'll be doing a Lambda within a VPC in a coming video. I'll look into incorporating RDS into it as well. Thanks for the suggestion!

[@Ckbagchi]

Loved the way u have explained the topic with example.

[@kelvintailor2448]

Great codding. Thanks, its really helpful to me also.

[@JesusAlfredoHernandezOrozco]

Great explanation. Thank you very much!

[@TomerBenDavid]

Awesome very clear 👍

[@rickfarina9835]

Clean and to the point... nicely done!!

[@BeABetterDev]

Glad it helped!

[@ViniciusFeitosa]

Thanks again. Your videos are much better than the official AWS documentation

[@BeABetterDev]

Thanks Vinicius! Appreciate the support.

[@krishnavenkatachalam985]

Excellent video.. I have a question - I see that in your cloudwatch logs then entire record or event got printed before every operation even though you did not explicitly print the complete record. How did it get printed?

[@cipher6042]

This video was dope and helped me alot with something Im doing for work, many thanks!!!

[@BeABetterDev]

Glad it helped!

[@harjos78]

Awesome!. Your videos are really very helpful.. Appreciate the efforts in making this video

[@BeABetterDev]

Glad you like them!

[@SamuelSantana1000]

Nice, Man! I was able to learn in few minutes! Thk!

[@BeABetterDev]

You're very welcome Samuel!

[@naveenkumar-jv6pc]

Thanks, Great Video on DynamoDB Streams.

[@BeABetterDev]

thank you naveen!

[@konstantinlitvin8071]

Very useful. Thank you very much!

[@BeABetterDev]

You're very welcome Konstantin!

[@renianx7610]

Thanks a lot. I believe they are excellent videos.

[@BeABetterDev]

Glad you like them!

[@jongschneider]

I love your videos. They have helped me better understand a number of AWS concepts. Do you use Terraform to manage your infrastructure as code? If so, have you considered doing a Terraform series?

[@BeABetterDev]

Hi Jonathan, I have just started dabbling with terraform. In fact, I am coming out with a video soon on AWS Cloudformation and how it compares to some other infrastructure as code providers. It should be released in a few weeks so stay tuned. I'll add your suggestion to my list of topics that need some love. Thank you for watching!

[@oerickmuller]

Great video, thanks a lot.

[@BeABetterDev]

Thanks Erick, glad you enjoyed!

[@malayalamtechvideos4473]

Good video. Thanks a lot.....

[@BeABetterDev]

Thank you! Glad you enjoyed.

[@PoojaSumann]

Thanks for the video. Please upload these videos with code in Java too.

[@pratikgupta9692]

Hi, such a great video! I just have one question. Can i do something so that it doesn't trigger the lambda for every event but does it on an hourly basis or based on the number of records inserted?

[@tanmayrane858]

Great tutorial !!! I want to display the lastest/newest row in my Table to the S3 Website, please guide.

[@salmanshaik452]

Excellent demonstration Kudos! How will you write the Modify Function if there are changes multiple columns in single record? Please let me know

[@venkataseshapyeddanapudi320]

great video. how about adding an attribute to an entry/row in the table and then to all the entries/rows at a single instance? Can you please make that video?

[@joeaabb]

Terrific series. Could you do a series on S53

[@seymabiskin8728]

Hi, can I trigger the same lambda from different Dynamodb streams? Then what would be the implications about sequencing as I want to keep ordering?

[@CptSupermrkt]

Love this channel. It's kind of interesting to me that CloudWatch Metrics shows up as a single invocation, but the CloudWatch Logs show three very distinct invocations with unique IDs, execution times, and memory usage. I was thinking that maybe with batching, the way Lambda works might be different than I'm used to, but according to the documentation, the RequestId that shows up in the CloudWatch Logs refers to "The unique request ID for the invocation," and your log shows 3 unique IDs, yet the CloudWatch Metrics invocation count is 1. Not really a question I suppose, just wondering how this is working under the hood.

[@ting-yuhsu4229]

would be interested, if you also show the handling when lambda failed to execute and send message to DLQ

[@SammyTvMan]

Great video

[@BeABetterDev]

Thanks!

[@soniauppal]

Excellent very usful video but i have a question when you said left side we have references insert on our DynamoDB table so this reference code is made by us only and where we are keeping it please help!!

[@windowcrystal4012]

Excellent video, You really have a gift of simplifying aws! But maybe I missed some parts, at 4:08 your stream enabled is "No", but at 18:33, it is enabled. Did you enable it manually by yourself and where I can find the reference?

[@BeABetterDev]

Thanks Crystal! Yes I think I must have accidentally cut out a step during editing. In order to enable the stream like it is at 18:33, simply go to your Dynamo table and click 'Manage Stream' under the overview tab. There you can select stream settings such as New Images, Old Images, or New and Old Images. In this tutorial, I used the following setting: "New and old images - both the new and the old images of the item" Hope this helped!

[@infinteuniverse]

How'd you get the test templates you have there on the right at 7:10?

[@kiranmahesh93]

Where is the video to check why try catch block necessary for lambda function in aws, please let me know as I was searching it from yesterday

[@devilangel036]

Do you have a video on how to read data from dynamodb stream using python?

[@nishitrathi4730]

Great explanation. I have one question, How do we send the inserted/updated/deleted rows to Elastic search ?

[@BeABetterDev]

Hi Nishit, glad you enjoyed. What you could do is parse the results from dynamo in the lambda function and perform a batch write to elastic search to index the data. I'll be putting together a video on this topic in the coming months, stay tuned!

[@mr-oe2kd]

Can we pipe dynamodb stream directly to aws eventbridge without a lambda?

[@javiasilis]

Thank you for the explanation. Thanks for taking the time and energy to perform a very simple and understanding tutorial. Now talking directly into a real application. What if I designed DynamoDB in a single table instance? From what I'm seeing DynamoDB streams do not differentiate the type of object that was inserted, unless I add a special key that identifies the type, which I can then use to filter it out. Without knowing I've been doing what DynamoDB streams offers in a manual fashion, by implementing Domain Events directly in the application. What I like about Streams is that your application becomes a little less susceptible to out of sync because you guarantee that an item is inserted. What I'm still thinking is that if your application needs to do a multi-step insertion, (User creates an account, then you need to contact a third party service, after that send an email... what if step 2 fails, and I can't let the user continue without proper account creation?) and one of those data fails down the line, you'd still need to look for a way either to recover or to rollback the operation (Dead Letter Queues I suppose, or letting the user know about the failed operation).

[@javiasilis]

Now that I'm thinking about it, DynamoDB Streams is an excellent Event Sourcing provider. If you persist the items with events, then you could potentially play them back in case of failures in subsequent requests (As there exists parent-child relationship in DDB Streams).

[@MCAMarshallS]

hii bro , can you provide aws rds , dynamo db lab videos link ?

[@makhus3478]

Can you explain how to push to elastic search (insert/modify) from dynamodb. Please take care of index while pushing as ES will automatically give index. Push index also from dynamodb (id)

[@abhnvjn]

Hi i'm looking for the same, have you found some resources for the same.

[@i.ankitmishra]

Thanks for this great video!! Can you make video on some specific group getting a push notification when someone from other group update/insert/delete in the dynamodb table? or if you can suggest a way to do that, it would be helpful.

[@BeABetterDev]

Hi Ankit, Thank you for the support! Regarding your question, can you explain what you mean by "specific group"? I may be able to provide some suggestions.

[@i.ankitmishra]

@@BeABetterDev Specific group means a user group like whatsapp where there is only one admin who can send message and everyone else are just receiver(and they will get notification)...basically one to many communication. Admin will insert data to DynamoDB and fixed set of users will get notified that there is a change in data. I think I can achieve it using DynamoDb Streams, Lambda and SNS. Let me know about your thoughts about this and any other way to achieve it. :D

[@priyankapatel9461]

Useful video! How can I achieve this requirement? Collect and store information regarding creation and deletion of S3 buckets and also creation and termination of EC2 instances in the AWS account 1. Create a CloudWatch Rule to listen to the below AWS services event sources and event types: a) S3 - Create and Delete bucket operations b) EC2 - Create and terminate instance states 2. The CloudWatch rule from #1 should trigger a Lambda function, the lambda function should parse the event to log the following details about the event in a DynamoDB table: Hint: Use AWS SDK for Python (boto3) to store the information in DynamoDB table. a) Event time b) Event source c) Event name d) Resource name (Bucket name or instance ID) e) AWS region f) Username

[@nikitasharma5087]

I did the same but I am facing an issue that some of my records are missing from the lambda event, i.e. if 100 records are being processed in dynamo then the lambda receives 97 records ... some records get missed, I can't find them in the event, anything I can do in this matter please suggest.

[@N1NJ4ASSASIN1]

Hey thanks a bunch for your videos, they're very helpful man! There's a new messaging system for CloudWatch, and it doesn't show the print statements from your code, rather REPORT, START, and END RequestId: followed by a unique id. Is there any way we can go back to the previous console to check our print statements and details within our lamdba functions? This will help out a bunch when I'm error handling my own functions

[@BeABetterDev]

Hi Kieran, You're very welcome! I didn't realize they changed the way print statements get outputted. Can you try adding a couple test lines to your print function to see if this is indeed the problem? Cheers

[@N1NJ4ASSASIN1]

@@BeABetterDev I tried that but found that you must attach a policy to your lambda that will allow it to put logging statements in cloudwatch. Otherwise it just goes through requests info and other stuff unrelated to debugging code.

[@sharadagarwal16]

That was very informative. 1 quick question tough. I see you never enabled the stream on your DynamoDB table. @ 4:12 you can see that shows "Streams enambled: No", so wondering how it write the data onto the streams without enabling that.

[@BeABetterDev]

Hi Sharad. Good point, I think during the editing process I must have clipped this step. But I definitely enabled the steams via the console in order to get this to work.

[@rajmishra6190]

Excellent !! could you please update this code to interact with Lex bot ?

[@roshanmohammad2969]

Thanks for the Healthy Session. I need one clarification, Now I have no permission to Create Role as per my Organisation.How can I still trigger dynamo Db using lambda. Steps I Performed: 1.Created Dynamo Db. 2.Created Lambda Function. 3.Created Trigger and Added the Lambda Function. 4.Now Creating Role throw Permission Error.

[@khilielbullock8509]

I wonder is it possible to do like a lambda to check to see if a name (or something else) already exists within a dynamoDB database. If it does, then... x, if not then... y

[@feelsveryChadman]

Hi , I had a question . Can you load each stream 'event' that you're iteratively parsing into a buffer and load that buffer into s3 ? If so then how ? Thanks a lot for the great content !

[@BeABetterDev]

Hi Ayan, Great question. You can potentially load this data into Kinesis firehose. Firehose has a buffer functionality where it will deliver data to S3 in periodic batches. Check out these two videos for more details: kzbin.info/www/bejne/eoG3ZKatl5utnas kzbin.info/www/bejne/i36un3Z7nKpgY80 Cheers

[@feelsveryChadman]

@@BeABetterDev Thanks for the instant reply !!I had a follow up question . In case i do not want to use firehose , and directly want to put stuff into s3 simply via my code , is there some module that can let me artificially create a buffer in my code ? I read that the s3 put api accepts a payload in the form of a buffer as well , but i couldn't make any headway .Thanks. My objective is to batch my dynamoDB stream payload into a buffer and put it into s3 following which an s3 to redshift loader will get triggered(the native copy trigger functionality).Any help would be greatly appreciated

[@BeABetterDev]

Hi Ayan, Hmm unfortunately I don't think that is possible using S3 directly. Sorry about that Ayan. If you figure out a way I'd love to hear about it though! Daniel

[@knaraya936]

Thanks for a good intro to dynamodb streams. I noticed that you did not enable Streams when you created the dynamodb GameScore table, yet you enabled streams roles in IAM which you attached to the Lambda function, and you used triggers in the dynamodb table and everything worked. So I am still confused about not enabling Streams - is that really ok?

[@knaraya936]

Oops looks like you have answered this question below...please ignore and thanks once again for a good intro.

[@BeABetterDev]

Thank you!

[@shoebmoin10]

What is the software that you use for making these videos?

[@BeABetterDev]

Hi Shoeb, I am using Obs for recording and Adobe Premier for editing.

[@brucedeo1981]

my Tables, how no option to set up triggering from there to the Lambda function. New console experience, contains no such option (unless i dont see it). Switching back to the old console i can see it.

[@brucedeo1981]

Also, i can in no way select DynamoDB in the output...

[@JackAbou2]

can you do a video on lambda and AWS load balancers. thanks

[@BeABetterDev]

Coming soon!

[@piyushmajgawali1611]

can I control event generation? I don't care about insert and remove. I only care about modifications in certain fields.

[@BeABetterDev]

Unfortunately no :(

[@yelloverz5537]

But where you put the code for inserting into DB

[@BeABetterDev]

Hi Gijo, Check out this video for how to insert in DDB: kzbin.info/www/bejne/qGqyhHmjdtF4fsU Hope this helps

[@naveenkumar-jv6pc]

Hi could you please prepare AWS kinesis with lamda vedio

[@BeABetterDev]

Coming soon!

[@naveenkumar-jv6pc]

Thanks

[@knandi73]

To make your videos carry better quality and professionalism, please avoid the points below. 1. Do not make sounds with lips after speaking few sentences as a pause-maker. 2. Avoid musical pronunciations.