Event-Driven Architecture: I do not think it means what you think it means

With the popularity of Microservices, Kafka, and Event Sourcing, the term "Event" has become pretty overloaded and has caused much confusion about what Event-Driven architecture is. This confusion has led to conflating different concepts leading to unneeded technical complexity. I'm going to shed some light on different aspects of Event-Driven architecture, such as Event Sourcing, Event-Carried State Transfer, and Events for Workflow.
🔗 EventStoreDB
eventsto.re/codeopinion
🔔 Subscribe: / @codeopinion
💥 Join this channel to get access to a private Discord Server and any source code in my videos.
🔥 Join via Patreon
/ codeopinion
✔️ Join via KZbin
/ @codeopinion
📝 Blog: codeopinion.com
👋 Twitter: / codeopinion
✨ LinkedIn: / dcomartin
📧 Weekly Updates: mailchi.mp/63c7a0b3ff38/codeo...
0:00 Intro
4:56 Event Sourcing
9:48 Data Distribution
16:43 Notifications
23:50 Tooling
Referenced Blog Posts
Don't Let the Internet Dupe You, Event Sourcing is Hard
chriskiehl.com/article/event-...
Event Driven Architecture - 5 Pitfalls to Avoid
/ event-driven-architect...
#softwarearchitecture #softwaredesign #eventdrivenarchitecture

[@tristanstricker967]

I love how the focus is on the concepts rather than the nitty-gritty implementation

[@KA-wf6rg]

Some of these concepts have been explained in Derek's previous videos, but I appreciate that he doesn't mind repeating himself. I've found it has helped me quite a bit hear things explained again, perhaps slightly differently, and reinforces my understanding.

[@CodeOpinion]

Yes, this video is a shorter version of a talk I did at a conference really. This video and the talk are really mix up of a bunch of videos I've done.

[@basilthomas790]

Excellent presentation once again!! 2 points I would add are: 1) Event Sourcing is the in order persistence of application state change events as they occur to be replayed later in order to rebuild application state to current state 2) Domain events should be used within your domain for internal workflow processing and Integration events should be used to globally notify any non-domain services externally Best point that I totally agree with: Event Sourcing is an implementation detail. Absolutely accurate and this should be clearly understood BEFORE trying to implement an Event Sourced system.

[@MrDomenic123]

I just love your videos. This video felt a bit like a summarization on your previous video. Keep up the good work 😊👍

[@CodeOpinion]

It was a summarization for sure. It was based on a (longer) talk that I did at a conference recently.

[@baseman00]

Wow so much ground covered. Really good watch. In the example for state transfer where Service A is requesting data from Service B but Service B has an outage something that might be discussing is Reactive Manifesto on responsiveness and what resiliency requirements are required to make sure data is retrieved (maybe helping to preventing your fat event scenario)... Fat events operationally eat bandwidth. But I liked how you indicated it's blurring intent. These problems compound when maintaining context in downstream services, keeping operations with msg versioning, I've even seen people use it as a reason to treat the broker like a db.

[@NathanWienand]

Yet another banging video! Thanks for doing these Derek, such a wealth of useful information!

[@CodeOpinion]

Glad you enjoyed it!

[@ilijanl]

Great video and good explanation of the concepts of EDA. In my opinion events that carry state are really tricky and should be avoided when possible. One of the reasons is that the order of processing those is almost always crucial, additionally those kind of events become really fat fast (as you mentioned). When you need some state from other service to do some business logic, use notification events to invalidate, fetch & update the local cache. When that service is down for some reason, fetch the data from the local cache and make some assumption about the "accepted" data staleness. Further optimalisation could be made by throttling the remote service calls, having ttl or using techniques like stale-while-revalidating. For analytics purposes like datawarehouses etc use ELT or ETL techniques, you almost never need realtime analytics and if you do, it probably falls outside analytics...

[@CodeOpinion]

Thanks for the comments. Agree'd, unfortunately I do believe at this time, EDA is primarily being used for data distribution.

[@haraheiquedossantos4283]

Congrats. Really good video

[@moger83]

A pleasure as always!

[@Tony-dp1rl]

I have never thought of Event-Driven as requiring Event Sourcing. I've worked on several Event Driven architectures, and none of them used Event Sourcing. Odd how terminology is different to different people.

[@f135ta]

Well, lets not forget who Dereks' video sponsor is.. He's bound to have a bias towards Event Sourcing.. But, I'm with you - event sourcing typically covers about 1% of most systems really.. Pub/Sub is by far the most popular approach in my experience. State carried events being probably the most common "sub" approach..

[@FacingBlocks]

But you can’t trust the client to sent correct data to the microservices so we can’t ensure data consistency in the databases. Example: What if the client tells Order service to put product A and B into the shopping cart while It also tells the Payment Service that he only added product A. On top of that we could tell the Warehouse service to ship products A, B and C. In this case we would only pay for 1 product and receive 3 instead. Adding the order details as payload to the event is the only solution I can imagine.

[@lee45283]

Great video! Super useful!

[@gosnooky]

5:04 - The SKU starts with "YYZ" - Toronto has entered the chat.

[@CodeOpinion]

That's exactly why I put it!

[@ItsGonnaRaiN143]

I'm starting developing an EDA architecture, I'm currently facing a concern about doing it asynchronously or synchronously because we have a backend for frontend service that is listening a queue (running on a single thread) to reflect any data change in "real-time", we don't have async code on our Python modules to consume, produce messages but if we would like to migrate it to async it would affect the data consistency because of the async processing nature, what do you recommend us to implement/investigate/review? sorry if you already talk about it in the video, probably I missed it but would be great some extra insights about this kind of issues. Thanks for this amazing channel!

[@NervousNab]

Is there any good book explaining EDA in details? If yes, then which one?

[@GackFinder]

My current client wanted to build an event-based architecture for some of their internal line of business applications. So for some reason they chose to build it on Azure Service Bus, and for some reason the events are essentially Event Carried State Transfer without capturing any domain intent, it's basically just the U in CRUD. Let me tell you... it's an absolute dumpster fire. I've never been less productive in my career as a consultant. We're always running into issues where we can't develop certain features because the architecture is actively preventing us, and if something goes wrong, it's almost impossible to answer what actually happened after the fact.

[@Bosslogq]

Thanks for the video and all the work you do. Lots of confusion is also around CQRS mabye that would be another interesting thing to explain. I am also working on adding event in one on my app. This is when some transaction is completed. The app which listens to this event is (the only app at the moment) only cares about this event only if the transaction country is France. I dont like the idea of adding if before sending the event, I think it should be on the receiver side. But my team argument is that it will generate a lot of traffic on the queue because majority transactions will be German transactions. Would be great to know what you think about case like this. Thanks

[@mgjulesdev]

The team argument is valid. You can make the publisher side configurable based on the origin country so that in the future you can painlessly accept other countries without any code change. But do realise that then you are back to square one if you add other subscribers as some of them will indeed receive events originating from countries they don't care about. So you still end up with having to implement logic to handle that on the subscribers side. It's really a balancing act. You can also have a special topic namespace for that app, event and country but then you get duplicate publishing logic.

[@rodrigoalves8554]

Excellent video!

[@CodeOpinion]

Glad you liked it!

[@szotsmiklos8549]

This is an incredible video and great explanation of the concepts. I'd love to discuss some of the topics in greater detail I see a couple of issues with the approaches mentioned in this video: 1. Often there's no UI after a workflow is started so we can't just shift the data distribution's task to the user. (the order service and payment service example). 2. Similarly , event notification requires every service/component in the flow to either already own the data needed or take it from the source via an API. 3. Regarding event carrier state transfer, we often see an issue with event ordering (needs to be just as good / solid as with event sourcing) but more importantly the consumer might depend on historical data which is harder to get in a non greenfield project. Do you have (other than your channel) any good resrources that could answer my questions?

[@CodeOpinion]

Check our Oskar Dudycz event-driven.io/

[@sirg7573]

Unrelated question, Derek. I joined your channel, but can't figure out how to access the private Discord. Can you let me know how?

[@CodeOpinion]

Scroll down to the part for Viewers. You need to associate your KZbin account with your Discord account. Ping me in discord if you can't get it sorted out support.discord.com/hc/en-us/articles/215162978-KZbin-Channel-Memberships-Integration-FAQ

[@kobac8207]

In which way is this choreography if one of the components (the client in the presented example) is the one that holds the business process logic (order of the calls)? What I know is that choreography has no central point for business process logic, but as we plug in the events to react to each other we get that implicit process as part of events reacting to events.

[@zikkrype]

When there will be EventStore demo? I've checked their videos on their youtube channel it they leave feeling that it could be better

[@CodeOpinion]

I'll add it to my topic list. Since they are a sponsor that may come across a tad weird but I'll see what I can do.

[@zikkrype]

@@CodeOpinion thanks, looking forward to it

[@evgenvb]

First of all, thank you for great video as usual and I must excuse for my poor english language skill. I have been confused by your example of orchestration doing on client, what happens with already written data on service "A" if client can not execute command on service "B"? I understand idea of workflow engine, like Comunda, as a core of business-processes management in distributed system, but client itself, or some type of smart-gateway - is not seems a good idea for me. May be you can provide some more detailed mention about it? I have build large distributed enterprise system with event sourcing and used ES events to distribute state, with sagas if i have need some consistency, of course i used BPMN engine to orchestrate some large-scale and configurable processes, but lots of contexts was coupled through ES events state distribution. For example: Employee has some attributes that have matter in salary calculation, we have context that manages Employee and attribution changes, and context of salary calc, attributes means slightly different for contexts, but are relevant in both. Tell me please what is wrong in this?

[@CodeOpinion]

"Client" doesn't have to be the end-user or UI. It can be an API gateway or orchestrator. As to the state being required in both, often times there's a difference in when that data is required or relevant. I should create a video with a concrete example of this. RPC isn't horrific all the time. It's when it's a free for all service to service which is. I'll try and create a future video with some examples.

[@evgenvb]

@@CodeOpinion i share your pov about client, but seems that "data" is "anemic" term, in my example data "means" slightly different in two contexts and must be changed in sync(in terms of business logic of course), how we can delegate to define this "rule" to a "client" event when "client" is orchestrator? May be we must define that "business level" or "domain level" has levels itself, and we have some "state level" and "process management level" as a "client" to one? ... and one more thing: ok, i have ES and it is beautiful in technical simplicity, we applied events on aggregate, and publish them, but this events propagate state and are "fat" by definition, how we can get "light" versions of events, must we implement some transformation pipe or some bus do decide what we must publish or what we must transform and republish? Thank you for answer, and will wait for future videos of course )

[@florianfanderl6674]

What if you had a second service that wants to know what's currently in the shopping cart (e.g. to make recommendations)? Would you still publish ShoppingCartChanged(ItemAdded/Removed/QuantityChanged...) events as event carried state transfer?

[@CodeOpinion]

Yes you could, but the data itself in the event would probably be pretty limited to the CartID, ProductID. Fat Events usually end up being entire entities or grow overtime when you keep adding data for specific use cases. This generally is also because it's CRUD driven.

[@florianfanderl6674]

@@CodeOpinion well I would design them specifically based on the needs of other teams. I'd rather not broadcast my entire entity with all its fields.

[@Bosslogq]

Does it mean you send different events per what receiver needs?

[@florianfanderl6674]

@@Bosslogq well normally you don't know the direction you're system evolves. So my default is to keep all data/events private and only expose the data/events that other teams need. It's the exact same pattern that I use in code.

[@WilliamPowerDental]

Any thoughts on Event Modelling? Seems to be the glue that binds the requirements to the implementation

[@CodeOpinion]

Yup, I should get Adam on here for a chat.

[@offbyjuan]

One thing that's not called out in the workflow choreography example is if the payment fails. How do you notify the client?

[@CodeOpinion]

Email. Oddly good timing because my credit card expired and I was issued a new card. That's exactly what I got when payments were failing for subscription services I had. "hey your payment failed.." emails.

[@offbyjuan]

@@CodeOpinion what about in scenarios where it's not okay to notify by email? An RPC sort of example

[@slowjocrow6451]

I like the concept, but how to implement it with code? I have a monolith and I'm looking at your example @22.22... I don't understand how to create this broker, and how the Consumer (Email) is aware of a new Order Placed event at the hroker. Really keen to learn how to code this

[@CodeOpinion]

It really depends on the tooling you're using. Most messaging libraries in the .NET space do all the routing and mapping for you based on type information and configuration.

[@ganeshanageshappa9341]

kafka - A distributed log!

[@bunnihilator]

This is GOLD!!! Just one thing. I have an e-commerce platform and a shipping platform. The first one have countries, states, cities. Wich i am sending their data to shipping. Which becomes a stale that. But that's with the purpose that the shipping platform should have that data to work even if ecommerce platform is not available. It cannot get that data through Rest Api call because that would make it dependant. So yea, there's a duplicate data so that the shipping platform is autonomous. Does this situation justify doing it? I believe so. What do you think?

[@CodeOpinion]

Sounds similar to my example about billing. Send the shipping platform the information it needs about an order during the workflow from the client. If you don't have the client do it, do it with asynchronous commands using orchestration.kzbin.info/www/bejne/nnjMgYqcjKxrndU

[@ilijanl]

One misunderstood concept of EDA and coupling is that coupling still exist, the consumer service needs to know about the produced event and it's structure. So the coupling is the other way around.

[@bunnihilator]

One thing I dont seem to grasp in Event Sourcing, in case of a bank account balance, to calculate the current balance i need to start from the day 1 when the account was opened to calculate until the current date!! Which may be years of data (saved events). How is that performant!! Am i missing something about Event Sourcing?

[@Bhaclash]

You can make snapshots of your state at a given moment, so you don't have to always recalculate your actual state from day 1.

[@evgenvb]

It is most popular question about ES ) Most popular answer: snapshots, you saved serialized aggregate state every N events, so in worst case you need to deserialize state and read N - 1 event Slightly rare answer: if you have ES, you have some CQRS, so you need to read aggregate only on state change, for read you have different fast representation Rare answer: read 100000 events by PK from one table is just stream, sometimes faster than read complex model with lots of joins. Performance is not a problem of ES. Event versioning is very complex problem, for example.

[@ivandrofly]

2:45 - 1 2:50 - 2

[@ptomcsi]

Thanks

[@CodeOpinion]

Thank you!

[@dinoscheidt]

8:15 yeah should have been called “State Sourcing” 😌

[@mathewmcloughlin8312]

The rainbow blog post has tortured me for years

[@CodeOpinion]

ha! ya, I did a video about it awhile back, which you probably already watched, but for any comment readers: kzbin.info/www/bejne/j2jGkmRjYqp-qck

[@DiogoBaeder]

A Canadian using a product example whose SKU starts with "YYZ"? I see what you did there! 😉

[@CodeOpinion]

🤣 I should of used YQG as that's closer

[@yootoobnoob123]

Thank you for another great video! Any chance you'd consider using a darker theme for your slide decks? No complaints about your illustrations, but the white screen is harsh on the eyes after 20+ minutes.

[@megaman2016]

He should have called it data state sourcing

[@whatthefunction9140]

You are describing block chain

[@CodeOpinion]

Ah, no.

[@raghuveerdendukuri1762]

Immutability is one of the core expectations from blockchain. Event sourcing pattern in the order of events is the way, offering the opportunity to replay the events to arrive at current state. Replication log concept in mysql database happens as so. Blockchain is the popularized reference of distributed ledger technology in crypto currency (bitcoin) context.

[@lucasterable]

8:10 "Event Sourcing is that we will store *facts* which have occurred" then, dear Mr. Young, why the heck did you name it event sourcing instead of FACT sourcing?!

[@kopiking352]

event sourcing is still bad as I donot understand what is fact? how fact is event sourcing but not state, domain event, or what is happening? on the other hand i also don't like the idea to use event sourcing to update state - that is only good for system state or low level state more for machine-level events but not for business-level events.

[@theradfactor9104]

Answering your 2nd comment. A state can be the "current value", "current condition" or "current status" of a business transaction, product, or anything business-related. This is not a technical state, but a business state. As a simple example, a business would want that EVERY TIME a product is sold, the amount available for the next sale is updated. "How much is left" is the business information and its value is what we'd call "state". Failing to keep the state/value correct means the business could for example sell more of the product than it actually has in the warehouse. Without using Event Driven solution, A LOT of work would have to be made especially around when errors occur.

[@ThugLifeModafocah]

This is exactly what happened with TDD... later on with BDD... It is anoying as fuck this semantic diffusion thing.

[@emonymph6911]

Are you aware of an IT terminology dictionary? It triggers me that you keep using terms I don't know the meaning of so I have to pause and lookup then continue etc. If it doesn't exist why the heck not?

[@CodeOpinion]

Sorry. That might be a good topic for a video actually. Specifically which terms?

[@walterclementsjr.5947]

it's called wikipedia. This video is intermediate-level and he's only mentioning jargons you should've known at that stage.

[@emonymph6911]

@@walterclementsjr.5947 That's not the point walter, where is the dictionary of IT? Every industry has one, you guys pride yourselves on documentation and can't even agree on common definitions.

[@emonymph6911]

@@CodeOpinion I've thought about it before replying it would probably be best to keep as its own playlist as everyone is on different levels. You could have a [general web related terms ep1], [front end terms ep2], [back end terms ep3], [devops terms ep4], [architectural terms ep5]... When I started learning python (which is the beginning to learning code now a days) we just learned to write scripts that ran the code but I had no idea what things like state, tokens, events-listening, etc mean when trying to move into JS world so I hated it and gave up on it. If you wanted to make terminology videos you have the advantage of being able to illustrate of a message flows between terminology1 and terminology2 etc and what each stage means. So they will always be relevant and useful. Application and transport layer terms and most important for being who want to become devs but aspiring coders don't know this as they just get sold coding bootcamps. Anyway now a days I gave up on being a web developer (it wasn't from lack of trying, it was lack of industry consistency that threw me off) and personally only focus on data, DBs and like learning about architecture as a hobby. This is the only IT dictionary I know of in the data world by a data driven organization. www.dama.org/cpages/body-of-knowledge which is great and accelerated my learning. The problem with Wikipedia is you don't know what you should learn or in what order and different websites have conflicting and often incorrect meanings for the same word. Another serious problem developers have in their teaching style is they say "don't do A B C, do D" this is awful for learning complex topics as your mind is subconsciously trying to remember 4 things now instead of 1. Just teach the right way once that's it. If someone does a catastrophic mistake they will very quickly realist it themselves, revert back and rethink the problem which needs to be solved. Wikipedia often does this as well or goes into deprecated history which is irrelevant to 99% of new devs, that wastes your cognitive load. Another problem is 'half taught' where someone will teach you authentication then after 45min end the video with oh but in production we would do it differently because this poses security risks but we don't have time to cover it here. Now that half taught auth was wasted effort that has to be relearned in a new way, leading to confusion. There have been so many times where I made a lot of progress only to realise oh woops I didn't know about some foundational knowledge that was never mentioned and had to take two steps back (had no idea packets had metadata and expire after being handled X amount of time even though I know knew about algorithms, it should be taught the other way around). Having all the terminology description properly before starting would prevent all this. In my opinion the most important words relate to any time a code block stores/passes/listens/morphs/ information in some way. I'm sharing this with you because I know you get the big picture of code which is actually workflow. Also know that more people feel the same way but they rarely speak up about it because some angry elitist will try to dismiss them as stupid. Hope that helps. Won't hold it against you if you don't make any videos on it but at least now you know how some people who turn their back on developing feel about the ecosystem not being clear enough when training newcomers.