Event Sourcing Core Concepts

What is event sourcing? Often, many terms are thrown around that can be pretty confusing. I will explain the core concepts and the terminology used around Event Sourcing so you can better understand what it is and what some of the benefits might be.
🔗 EventStoreDB
eventsto.re/codeopinion
🔔 Subscribe: / @codeopinion
💥 Join this channel to get access to a private Discord Server and any source code in my videos.
🔥 Join via Patreon
/ codeopinion
✔️ Join via KZbin
/ @codeopinion
📝 Blog: codeopinion.com
👋 Twitter: / codeopinion
✨ LinkedIn: / dcomartin
📧 Weekly Updates: mailchi.mp/63c7a0b3ff38/codeo...
0:00 Intro
0:33 Events
2:01 Event Streams
4:12 Projections Read Models
6:47 Projections Write Models
8:00 Subscriptions

[@VladyslavHorbachov]

Hey Derek, thank you so much for covering this topic. This is the best Event Sourcing channel on KZbin. I have some questions related to implementation, and I can't find the answers anywhere: In case of Clean Architecture + DDD project, we have some layers, like Domain, Application, Infrastructure and so on... These projections should be part of Domain project, or some interfaces has to be defined in the domain, and the implementation lives in some other place? I understand, that the implementation is subjective, and you can do it in hundreds of different ways, but I want to heat your experience on this topic.

[@bernhardkrickl5197]

We do something similar. We do message exchange with customs. We treat the messages as events and project a series of messages to derive the current state of a declaration process each time a new message comes in or goes out and store the result in our relational data base. I hesitate to call it event sourcing, though. Because imo an event should be something "small", ie. it should not have much data attached to it, and it should be fairly flat. The customs messages can have hundreds of fields in a structure several levels deep with repeatable parts. It's not just "Customs event X has happened, changing the important bit of the state from Y to Z", it's "Customs message X was sent/received and here is the full data of it. Go figure it out yourself."

[@pierre-antoineduchateau923]

Great video as always. I think that it goes a bit farther than 101 though 😉. I would have loved to hear the pros/cons of event sourcing vs normal CRUD with tables !

[@nighma]

Great video! Thank you! Another interesting one will be how to manage atomicity for example when a projector need to write to the read store AND the event store. Another example will be when we need to write data into a database and send a message in a Kafka topic.

[@sykvel]

Projection only read existing events. New event could be created by aggregate object only that orchestrated by command processing engine. So depending of your commands you can apply different strategy - execute next command in bundle only when first finished successfully or rollback all events when one is failed

[@vincentcifello4435]

Marten has Inline projections that write to your read model in the same transaction as saving the event to the store. Although, can't you create an in memory projection is you need such strong read consistency?

[@MrDomenic123]

Is it just me or are others also surprised by the quite negative comments about event sourcing? I didn't expect event sourcing to get such a rejection...

[@CodeOpinion]

As with many that are unfamiliar or done incorrectly they tend to throw the baby out with the bathwater

[@capability-snob]

Another important lesson is: store the whole event. You're not using those fields? Don't care about that relationship? If you store those details, you can add a dependency on them later, replay your stream and have that detail as if you were always tracking those fields. Event sourcing is the ultimate schema update hack.

[@pierre-antoineduchateau923]

Exactly ! I'm starting to use it to kickstart early-stage projects, where the business logic is still uncertain, but the app is already running in production. By storing wide enough contexts in the events, I know I'm not losing information. I can change queries without data migrations. At that stage, i'm not even using projections. Those come later, to optimise performance.

[@marcom.]

One of the things that really sucks in event sourcing is the fact that we usually don't start to build a system with all its data from scratch. So you need some kind of special initial load treatments. And the second thing that sucks is the fact that event schemas usually evolve over time and you get more and more attributes that weren't included from the beginning. So you tend to make events very broad so you end up in some kind of event driven state transfer.

[@serb1146]

Event versioning, upconverters, migrations, performance, when u have 5+ years project support... Nightmare.

[@sykvel]

What the reason for having those broad event? It could be a new event or composition of objects that you include or not in event

[@sykvel]

For initial migration good practice is having init event for each aggregate to set state

[@lkazzzz]

You don't need all historical events about an aggregate before starting to apply event sourcing. All you need is to create an initial snapshot record of your current aggregate state, and then apply events afterwards.

[@afailable]

​@@serb1146 it's a trade off, because all the other options become nightmares for different reasons by the time you're into supporting a business system for 5+ years

[@awright18]

Good video! I've not yet done event sourcing in practice, but I have a question that I don't believe is typically talked about. It seems that projections "replay" events in order to derive the state, and I believe that replaying of events has to happen in code, so what happens when the shape of the events changes through different versions, it seems to code would need to be kept up to date to handle that problem for the set of data streams to remain "replayable". This isn't likely too much of an issue with a set of recent transactions, but seems like it could be a problem when wanting to surface historical data (several years old).

[@michaldivismusic]

I'd like to know as well.

[@pierre-antoineduchateau923]

The past events are not supposed to change shape. Think of the event history as un append-only log. New events can have a different shape (ie different properties) but the old events in the history should not be overwritten. Projections have to be updated to handle new event shapes but have to still handle the old events. You can give the event payload a version number to help the code handle different shapes. That's the basic principle but of course, you can use a different strategy.

[@michaldivismusic]

@@pierre-antoineduchateau923 So in summary, try not to introduce breaking changes to the event structure, but if you have to, use versioning. Sounds good.

[@RaVq91]

@@michaldivismusic versioning of events sounds bad, you would have to keep adding new events instead of modifying old ones when you would like to add some extra properties to them OR do some branching in "apply" method. Who will know that in some long living storage we keep some events that not match current event structure.

[@sykvel]

All your architectural mistakes baked into events consistently. Eventually or not:)

[@FlaviusAspra]

ES is what you do together with CQRS, in order to really profit from ES. Systems in which you do ES without CQRS are systems not complex enough to have been following ES in the first place. Zoom out. The two are ombilical.

[@CodeOpinion]

I would say they are _often_ go hand in hand, but you there are absolutely use-cases of ES without CQRS or projections.

[@dasfahrer8187]

Event sourcing just sounds like CRUD with extra steps that also just doesn't scale well as time goes on as you have to now potentially replay millions, billions or more transactions. So, to alleviate that overhead and time, if you're going to store "projections" then why do event sourcing at all? It seems like the better solution in most instances is to just do a typical relational database and then add a thin event layer on top for possibly some context, support, debugging and reporting.

[@CodeOpinion]

Using a series of events as way to persist how and why you got to a current state is natural in a lot of domains. If you try and apply CRUD thinking with event sourcing, you're going to end up in a horrible outcome. Most call this CRUD-sourcing. There's no point. The value is in when you're task-driven and in a domain or subdomain where recording the series of events that you can derive state has a lot of value. An example of this, while maybe not intuitive that I use in a lot of examples is inventory. You can project your GL from all of the events that happen in a subledger (invoices, receipts, etc), which is event sourced.

[@majormartintibor]

What is Event Sourcing? Event Sourcing is architecting for tomorrows questions.

[@sakesun]

Architecting for tomorrows troubles.

[@obiwanjacobi]

Of course product YYZ987 should have been YYZ2112... 😅

[@vincentcifello4435]

Thanks for the continued high quality information! Have you done a video about event sourcing and EDA? Something along the lines of a Process Manager/Saga. NServiceBus has that Saga integration tutorial. How do you see that working in an event sourced application? In several presentations, Adam Dymtruk states no Sagas or buses are needed. kzbin.info/www/bejne/eHnPfIOdjperrsk Thanks!

[@CodeOpinion]

I think I can see the angle, however I don't agree. Event Sourcing used as a way to record state and using them as notifications for workflows (saga) are very different things to me.

[@vincentcifello4435]

​@@CodeOpinion Either my question was terrible or I am thinking about this incorrectly. Wouldn't it still be necessary to have workflows triggered by business events (messages) in an event sourced system?

[@CodeOpinion]

No, you could be event sourcing and not using workflows. You could not be event sourcing and using workflows driven by events. One is about state the other is about communication

[@Tony-dp1rl]

Almost Nobody needs Event Sourcing. Literally < 0.1% of systems would even benefit at all.

[@CodeOpinion]

My example of inventory is naturally event sourced. Your sub-ledger can pretty much generate your GL as a projection.