How we use DynamoDB streams to replicate data into Elasticsearch

[@Nova-ul3vv]

Dude i feel ya! 😅 DynamoDB is nice but its hell to serve data to other applications. Especially if you have a lot of data which needs to be paginated on complex queries its simply not possible. We currently query all data then do further processing in lambda and put the results in memory. But your video gave me an idea how to restructure. Thx!

[@ImranKhan-xn6ig]

Crisp and Clean Explanation

[@renanlisboa123]

Why not use a SQL database? I'm not judging or anything I'm just trying to learn so I can make some decisions in my job.

[@WebDevCody]

i wish we did. I'm not a fan of dynamo unless you have "super large data" needs.

[@MdRahman-ho8fl]

excellent. brief and clear.

[@raygan3]

I dont know much about elastic serach but i wonder why do you need dynamodb, cant you just use elasticsearch as your database?

[@WebDevCody]

bonsai.io/blog/why-elasticsearch-should-not-be-your-primary-data-store

[@uzair004]

It would be expensive process. I need to know how I to improve dynamodb modeling and mapping process, it took me so much time to come up with few access patterns.

[@pshaddel]

Nice video! We are planning to do the same thing with Postgres, searching for phrases is not that efficient specially in our architecture that we have to join 4 or 5 tables. I am wondering if we could do it smoothly inside the AWS Lambda. My plan is to join all of those tables and put them as just one document in Elastic Search, which makes it quite fast, but on the other hand I have to update these elastic search documents in cases that somebody updates another table(let's say tags).

[@WebDevCody]

Elasticsearch is also a pain. Honestly id try to just find ways to achieve this using Postgres. In elasticsearch, joining data using parent child relationships is aweful performance, and also you have to keep creating new indexes the moment you change search properties or add new search properties.

[@pshaddel]

@@WebDevCody My plan was to denormalize these documents in Elastic Search, you are right, joins in elastic search would not give us the performance we want.

[@PTBKoo]

Is there a service that can provide similar functionality as dynamodb streams for planetscale mysql?

[@ebukaume]

Nice design, but perhaps we could buffer the stream using an SQS (with a DLQ) so the Lambda doesn't choke and so we could recover from errors in the lambda without losing data. WDYT?

[@WebDevCody]

I think the dynamodb streams (kinesis) is built to hold a LOT of messages, so I don't think it would ever choke. Also the lambda processing the streams from the dynamodb stream already have a fixed limit on how many message they can read per second, so I don't think they will ever choke per say.

[@ImranKhan-xn6ig]

@@WebDevCody Lambda could fail cause of other unexpected reason. Don't you think adding a SQS as buffer (with a DLQ) with ensure data processing of each event streamed from dynamoDB. Just curious can we ensure 100% processing of streams without involving Buffer SQS but including DLQ

[@carlosmedinaio]

I think a SQS (with DLQ) should be added for example just in case the communication between Lambda and ElasticSearch faces some unexpected issue, and the data record to be inserted is lost. Having the SQS, the message involved in the issue will go back to the SQS and event will be launched again. Am I ok?

[@SeibertSwirl]

Good job babe!!!

[@allenchun48]

What if I update, or delete a record in dynamodb? will it also reflect in Elasticsearch?

[@WebDevCody]

Yes if you handle the update or delete event in the stream, you can also remove it from elastic search.

[@adriancoliba]

Great, thanks!

[@vaultek_]

Where we get that dynagrams images

[@WebDevCody]

Which one? You’d need to take a screenshot

[@vaultek_]

@@WebDevCody I find it, no worries thanks

[@sheldonfourie5959]

Nice🤓