Event Sourcing: Rehydrating Aggregates with Snapshots

[@mrjamiebowman1337]

I really like that your videos are short, concise, and packed with information.

[@CodeOpinion]

Thanks! Hope they're helpful.

[@leandrogalvan1110]

I like that you mentioned the example with Orders being kinda short-lived. Whenever I picture event-sourcing in my head I can't help but think of really long-lived stuff such as Users, and say 'man, am I really going to reconstruct this thing from like literally thousands of events?'. Good insight :)

[@CodeOpinion]

In my experience, most are finite and have a limited lifespan/ that doesn't contain an overwhelming number of events. Again, just the domains I've lived in likely.

[@kevinohara80]

This is one of those aspects of event sourcing that’s usually mentioned, but not explained. Thanks for the concise explanation!

[@CodeOpinion]

Thanks 👍

[@aaronderenard3527]

Mr. Comartin says "Thou shalt not add 10!" Must add 11, or we might be children about it. haha

[@fredimachadonet]

Great video! Thanks a lot. I've just joined the community as a developer.

[@CodeOpinion]

Welcome aboard! Thanks for the support. I appreciate it.

[@andy_lamax]

Never mind my question from projections video. This one answered it

[@CodeOpinion]

Glad you found it!

[@codewithkashif]

Super awesome and realtime content! you never let down your standard, instead it is going next level in every post! Thanks a for helping!

[@CodeOpinion]

Glad you enjoy it!

[@F2H16]

Thanks a lot for all the content on event sourcing. Quick question, is there a chance that you could walk us through the comparison of Marten DB and EventStroDB ?

[@CodeOpinion]

Good suggestion.

[@codewithkashif]

Hey just curious to know that what next is coming in this wonderful series of event sourcing. Especially I am looking for a solution or workaround for - "What should we do with persisted events when that object structure is going to change" like for example earlier we had one "Name" property but now we have to segregate it in "FirstName", "MiddleName" and "LastName" or vice-versa

[@CodeOpinion]

Sounds like versioning is what you're after?

[@codewithkashif]

@@CodeOpinion Yes you can say that it is kind of versioning or may be weak schema..do you have any plan to publish any new video around these topics

[@codewithkashif]

In past we had faced similar issues with CQRS approach where we first persist data in relational db and then we also saves several other copies in elastic search just according to UI. so it makes read quite fast. However later in case of any major schema change in relational database makes all saved objects in elastic search obsolete. So in such scenario we used to keep running some kind of background/console process to refresh the elastic search data from latest relational db. It was kind of pain though.

[@robert5763]

Great content! Really like this videos where you show some implementation of the ideas you talk about. I have some questions: - Where should Saga state be stored when you use a message queue and events handling is async? For each event, a new thread/process is created and the Saga state has to be recreated up to the point after the latest event was processed. - I want to start a new side project and I'm wondering what would be the appropiate architecture, the app will be similar to Uber/food delivery apps with real-time map/chat/notifications etc.

[@CodeOpinion]

I'm not sure I'm following in regards to the first question in regards to sagas. When you have a long running process that uses a saga, it's internal state is most often times handled by the messaging library. You could deal with it yourself, however each inbound message that your saga is handling would need some correlationID so it could get/save that state to some data store. But most often times that's handled by the messaging library if it supports sagas. By chance have you watch this video? kzbin.info/www/bejne/qICcc4upoZl0g7M

[@robert5763]

@@CodeOpinion I'm going to watch it now. But yes, I was thinking about managing the sagas manually, using a correlationID makes sense, I like the idea of having an identifier for every long process. I want to play with CQRS and Sagas in Typescript, but there aren't many messaging libraries like, for example, in .NET. Thank your for your answer!

[@dsindun7224]

@@CodeOpinion is CorrelatonId about a user story ? In my opinion ,correlationid is used by many connected events .

[@ZinchenkoOleg]

Great job, thanks! I have a question: What if we have to add an event in the constructor (for example Product Added Into The System)? How do we restore the state of the Product model from the event stream without a snapshot? If we use a constructor we will add the event on restoring from the state. To restore a product from the saved state we can use reflection. But if we don't use snapshots? While writing the question, I've realized that we cant use the initial event to create the Product object using reflection, then apply other events.

[@surenot-ky6sy]

I am just curious, why should we build the current state of the product by expensive computation of events and then struggling of the expenses making snapshots and then managing these snapshots? It seems more simple way is to store current object state and have a log of events with the same explanation of the reason of changes as supplementing part of the main object. Anyway most of the time we need the current state of the object and rarely path how we get there. So im my opinion it is more greenish way to waste heat of the computation power.

[@CodeOpinion]

You could store current state/snapshot in a separate stream and always start there when building up an aggregate. However it won't be your point of truth, the event stream would be. As long as your event stream is the point of truth, the rest are implementation details.

[@TheSydBarrett]

Awesome! This video is very helpful for understanding shanpshots. Btw, is there any built-in or additional extensions/helpers for working with Snapshots? Maybe some basic abstractions for avoding this Position calculation routine.

[@CodeOpinion]

Nothing built-in that I'm aware of. Likely something you'd probably create and tailor to your needs that you could likely re-use.

[@rimbik1]

super ❤ concept clear like water.

[@CodeOpinion]

Thank you 🙂

[@rimbik1]

@@CodeOpinion solace eda summit is live

[@brendonvandoornum6123]

Great video, I like how the snapshots are just another event stream. I am curious, how do you handle the scenario where the ShipProduct quantity is greater than the available product quantity?

[@CodeOpinion]

As usual "it depends". Depends if that command can fail or not. Unless you're going to (pessimistic) lock a record at a database level, you could ultimately end up shipping more product then is available. So if it's going to happen, you're likely going to create some use-case for if it occurs. The resolution to that might not be complicated and simple as a notification/report. Check out this part of a video I talked about if commands can fail: kzbin.info/www/bejne/pXvMhp2fjMqYqbM

[@KonstantinKuda]

Great video. Thanks a lot! Which strategy do I have to choose to determine how often I need to do a snapshot?

[@CodeOpinion]

Snapshots are an optimization. I'd first be making sure I'm designing streams that are finite (usually). I'd only be going down the snapshot route if it was required because streams were taking too long rebuild an aggregate.

[@juhairahamed5342]

have 5 instances of account microservice which transfers the money from account A to account B and then updates the data in the Postgres database. My problem is A user sent five requests to the account service and all of my microservices are working in parallel each request went to all 5 services way but now the user doesn't have enough balance in the account and I am already checking if the user is having enough balance or not. but after 2 requests user doesn't have enough balance so I am in confusion how to check this and implement data consistency first before the request goes to another instance of the same microservice. How to get consistent currentQuantityOnHand value for different instances ..

[@CodeOpinion]

You need to handle the concurrency. If you're talking specifically about event sourcing, most event stores have optimistic concurrency. I show EventStore as an example here: kzbin.info/www/bejne/aH_OZJ55eN6oaaM

[@raghuveerdendukuri1762]

In an event sourcing & CQRS based system, we will use different projection criteria to create different read models and update them using event subscription. do we require aggregate in command side to depict current state in this scenario in addition to event store wherein, the same events will be subscribed to and are used to update aggregate in command side while read models are updated in query side of CQRS using another subscription to the same event? please share your inputs @Derek, thank you

[@CodeOpinion]

Your aggregate will replay the events from a stream (or from a snapshot + events since) to build up a projection used in the aggregate if it needs to check any invariants.

[@acegame1452]

First of all thank you for all the videos you make. They are amazing and very much needed in today's ecosystem. Second, is it possible to get the codebase for this video. I am building a new microservice and I would really like to implement the concepts of this video. If it is possible for you to share, please let me know. Thanks

[@CodeOpinion]

All the source code is available to my "Developer" channel members. Check out for more info. kzbin.info/door/3RKA4vunFAfrfxiJhPEplwjoin If you don't have the JOIN button, you can support me on Patreon. www.patreon.com/codeopinion

[@acegame1452]

@@CodeOpinion Sure, I just did. Just so you know I did not support for the source code, but for the fact that you have been really helpful for me. Thanks again

[@CodeOpinion]

@@acegame1452 I really appreciate that. Thank you.

[@acegame1452]

@@CodeOpinion one last thing. Can you please help me out for the topic I reached out in patreon messages.?

[@AlfonGeek]

do you create a stream for each Product sku ?

[@CodeOpinion]

Correct

[@raghuveerdendukuri1762]

Can we consider that aggregates are maintained to be able to record snapshots when required while also being useful to do business rules validation of the data inputs? Please also share if we can access read model from command handler or if it violates CQRS, do we need to access through a different scope?

[@CodeOpinion]

Aggregates are consistency boundaries ultimately. Check out this video: kzbin.info/www/bejne/bGXRmINjlrqMhsU As for using a read model in a command handler, it depends why you're reading against it? If its to perform some validation, the moment you read/query it could be stale and something change it. Hence why you would want an aggregate.

[@raghuveerdendukuri1762]

@@CodeOpinion thank you

[@dsindun7224]

Great job . I have a question that how do you persist snapshot object , in a realtional db table ? is Snapshot an event ,like productshiped?

[@CodeOpinion]

Generally store the snapshot is just another event but often stored in another stream.

[@dsindun7224]

@@CodeOpinion thank you !

[@rustamhajiyev]

Awesome video! I have a question: let's say I have a domain event and the related Apply method have a bug. After couple wrongly applied events, the final aggregate state is incorrect. At some point I fix the bug and when the events are replayed I get the correct state but from a user perspective that probably will look quite weird: how the value changed without an interaction. What are the best practices for this kind of situation to both fix the bug and don't frustrate the user. Thanks!

[@CodeOpinion]

I don't see it as being different then any other bug really. Let's say you were using RDBMS and you were doing a SUM() on a set of rows. However there was a bug in your WHERE statement that you fixed. Now the result of the SUM() is different and is shown correctly to the user. Would you do something in that case?

[@rustamhajiyev]

@CodeOpinion, thanks for your answer. That makes sense. I'm just trying to figure out potential problems I could have with EventSourcing. Recently I watched a video kzbin.info/www/bejne/fHyppmpuj6p8n5o where Greg Young is talking about cons of EventSourcing, and one of the problems he mentioned is versioning of domain events. Do you have an experience with it? Some techniques you would recommend? Maybe that would be a good topic for another video in this series. Again, thank you a lot for the great content!

[@CodeOpinion]

@@rustamhajiyev Ya recently watched that video as well this week. I actually left a comment in regards to the Q&A portion. I think what Greg mentioned in regards to it being "different" is the biggest issue people have and not knowing the various problems/solutions that come with it.

[@ghevisartor6005]

Ok so i get concetps of the video in general, but does anyone know where can i start to learn about these topics? Like a starting point. Is it all from DDD?

[@CodeOpinion]

Event Sourcing and DDD are two different things. Check out this video if you want an overview of what Event Sourcing is: kzbin.info/www/bejne/d4bNZYBjqNlmn8U

[@ghevisartor6005]

@@CodeOpinion thanks a lot !

[@JamesGardner159]

Great video!

[@CodeOpinion]

Glad you enjoyed it

[@asdasddas100]

I imagine this wouldn't be as useful for an application like Twitter? Or am I mistaken?

[@CodeOpinion]

Well depends how you think of Twitter from the trivial perspective of tweeting and a feed. From that perspective, probably not.

[@dansharpe4535]

great video as always!!! have you used EventStoreDB in production at all? and if so, what are your experiences with it in terms of hosting/scaling etc?

[@CodeOpinion]

I have in the past, but not recently. The biggest issue was I did not want to have to manage a cluster. I wanted to have a managed service. Having said that, they do now have an cloud offering. Worth checking out.

[@dansharpe4535]

@@CodeOpinion have you used anything else or have any recommendations?

[@CodeOpinion]

@@dansharpe4535 If you're using a relational database, you might want to look at Marten or SqlStreamStore.

[@dansharpe4535]

@@CodeOpinion one last question, would you have an event for initial creation of the object (ie when a new product is created) or do you just use the constructor and apply one of the events then save?

[@CodeOpinion]

@@dansharpe4535 There could be a specific event or many that must start at the beginning of a stream. It really depends on what it is. Eg, a warehouse product might start by ProductReceived. It can't be started by ProductShipped, since there isn't any yet.

[@coffee.coyote]

Great video! Could you show us also how to create efficient search feature over current state of aggregators?

[@CodeOpinion]

Not entirely sure what you mean? Could you clarify a bit more?

[@coffee.coyote]

@@CodeOpinion so e.g. in our app we have products that have a price property on them. Our store has a wide range of products (thousands or even millions). The price may change over time. As a customer, I would like to find products ranging from $ 20 to $ 30. To do this, I need to find a snapshot on each of them and apply all events higher than the snapshot. Is there a more effective way to do this?

[@CodeOpinion]

@@coffee.coyote Check out my video on projections! kzbin.info/www/bejne/mIW1m4Jsf7BnjdU

[@coffee.coyote]

@@CodeOpinion Awesome, thanks

[@aruns78]

How would the Snapshot handle/correct for out of order events or late arriving events?

[@CodeOpinion]

Not sure why you would have out of order events in your event stream?

[@aruns78]

@@CodeOpinion The event store may be built to support an edge application. Think IoT, network latency or upstream application outages can cause late arriving and out of order events. Straightforward replaying of events from the event store may not be impacted if aggregates take into consideration event time stamps and starting from event 0. If snapshots are sequenced based on version#, the complexity is with correcting snapshots. A thought is to write 2 snapshots every 1000th event; one for 500 and one for 750. Build aggregate based on n-2 or 500. Have you had a need for this or something similar and had some solution?

[@CodeOpinion]

@@aruns78 I'd be interested in if you can get the event stream to be in the correct order by using a type of defer?

[@aruns78]

@@CodeOpinion Thanks, that is some to consider. BTW, thank you for posting these videos, it's great material and very well presented.

[@JeffChentingwei628]

But it still has many versions of snapshots...

[@JeffChentingwei628]

How often do you make a new version of snapshot ?

[@raghuveerdendukuri1762]

The version thought process that is mentioned in here is to identify the event upto which data is used to create the snapshot. On other hand, Greg Young in the following url (kzbin.info/www/bejne/fHyppmpuj6p8n5o) suggested to use versioning thought process so events can be versioned and thereby data transformation to be applied so data becomes compatible or near compatible to the most recent event version. In cases when such attempts are not possible, then he suggested to maintain both copies as a better possibility. please share your views on versioning of events especially in scenarios when we cannot take systems down for data transformation kind of...

[@CodeOpinion]

I'm talking about version as an ordered number of events in the stream. Not in versioning in the sense of making changes to the event types, similar to versioning an API. I will cover event versioning in future videos.

[@raghuveerdendukuri1762]

@@CodeOpinion agree and thanks