Event-Driven Architecture | Event-Driven VS Request-Driven Architecture, When Not To Use Events

In this video, I will be comparing event and request driven architect, describing the pros and cons of both, so that you can make better-informed decisions.
Also the following book is a MUST read amzn.to/3mDuBHD Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems (this is a book which I'd recommended for every technical person and has had pride of place for many years on my desk)
This video is NOT sponsored. Some product links are affiliate links which means if you buy something I’ll receive a small commission.
In this video. I will be explaining all this by running you through two different approaches for implementing a user registration process where I will be sharing the trade-offs between the two different architectural patterns.
See the written blog post version here: danieltammadge.com/2021/01/ev...
#eventdrivenarchitecture #danieltammadge #microservices #requestdrivenarchitecture
I have gained my knowledge by watching talks and reading many articles, but more importantly designing, and running event-driven systems in production for many years, along with making many mistakes during that time.
Designing Event-driven systems by Ben Stopford (This is the best book I have read on event-driven architecture) www.benstopford.com/2018/04/27...
Now a great article on event streaming is an article called "Turning the database inside-out with Apache Samza" www.confluent.io/en-gb/blog/t...
And I would also recommend the book Building Event-Driven Microservices by Adam Bellemare Building Event-Driven Microservices amzn.to/3pz2QOS
Following my experience with event-driven systems, I found Martin and Ben's content to align with what I have experienced and learned, while there are some viewpoints in Adam's book which I don't agree with, it is an excellent book and strove of knowledge with many nuggets of gold in its pages.
Hope you find these books and articles as useful as I have in your journey in event-driven architecture. Just FYI the amazon links are affiliate links (#CommissionsEarned).
-
I use www.lucidchart.com for my diagrams & www.flaticon.com where I use my pro subscription to find images for my content

[@HimanshuGupta-nk9yf]

Great video Daniel! Straightforward and with useful graphics. Loved it!

[@Danieltammadge]

Thanks I am glad you found it useful

[@Danieltammadge]

If you have questions or want me to drill down on a certain topic or pattern in the future then please let me know in the comments

[@futuristcorporation]

This video cleared up a solid bit of the confusion I've been feeling about the difference between REST/SOAP and message brokering. Thanks for making and posting! What I don't understand yet is how event-driven architectures avoid the possibility of losing messages between the client and the input queue or between the completed events topic and the client awaiting the response.

[@Danieltammadge]

Thanks Matthew for watching I’m glad it helped. I see 3 approaches. The first is if you are having to use distributed operations, what I mean here is if your service consumes a event and performs operations against a database and then publishes a event and then only commit or acknowledge the message. But in this approach the process needs to be designed in a way that if you cannot publish a message then you can reprocess the message again and either the operations you have used is keeping state and prevents the already performed actions from triggering again and only perform the actions which were not called, like publishing a message. You also need to ensure that downstream services has logic to also guard against the possibility that the upstream service may publish the same message more than once. The second approach is that you break the solution down into cell or functions, but each one is triggered by an event. For example when you get a request, you don’t perform logic or store the request, instead you publish straight to an event bus and then have your processor subscribe to the event bus. A good pattern for new green field solutions But what if you have an existing solution system, then the third is to use the outbox pattern which works for a database operation, the processor in this case does not publish a message after performing a db operation but instead as part of the database operation it inserts a record into a table which if hopefully running in the cloud and the database is a cloud vendor solution then hopefully on a record insert you can trigger a lambda function and then that lambda function publishes an event. If you are not using a cloud database with that capabilities then you have options of using a CDC tool, native to the database or a third party like Kafka connector or a open source tool like debezium.io. The third option is the one I would recommended for a solution when you are introducing events into a already live system. This option has only actually been. Available to the masses in the last couple of years. Before hand if you wanted the third option, you would have to developed a home brewed database change solution or purchase an expensive service like Oracle Golden gate (if you are using oracle)

[@MarcoLenzo]

Thanks for the video Daniel. I love the topic and it was nice to see the way you explain it!

[@Danieltammadge]

Thank you Marco for taking the time to watch and for leaving a comment. Glad you found the video useful

[@DebonaireBlkMan]

Thoroughly enjoying this series. The event-driven archtiecture/ stream-driver architecture knowledge space is still very new. Looking forward to more videos

[@Danieltammadge]

Thanks Nick. I am currently in the process of creating more so stay tuned

[@Danieltammadge]

Hi Nick any event-driven architecture topics you are specifically looking into? Or problems you are looking to solve?

[@DebonaireBlkMan]

@@Danieltammadge "kafka as an event store", "versioning events", "event-sourcing vs cqrs", "kafka alternatives", "event driven architecture"

[@loganpowell]

Thank you for being one of the only videos on event-driven architectures that doesn't suffer from shiny object syndrome

[@Danieltammadge]

Thank you, Logan, that is one of the reasons why I created this video. Too many people are leaping onto the event-driven train without understanding the journey they are committing to.

[@stasthesauce4641]

bruh your vids, are hecka underrated, thanks again for learning and taking this time to do this.

[@Danieltammadge]

Thanks for taking the time to comment

[@prashanthb6521]

Excellent, Thanks a lot Daniel, lucid explanation !

[@Danieltammadge]

Thanks I am glad it helped you

[@charlesopuoro5295]

Thank you Daniel. I definitely found it useful.

[@Danieltammadge]

Thanks Charles for taking the time to comment. Glad you found it useful

[@kristinaribena1654]

Loving the vids

[@snehotoshbanerjee1938]

Great video Daniel!

[@Danieltammadge]

I’m glad you liked it and thank you for taking the time to comment

[@thomas-sinkala]

Great video as always

[@Danieltammadge]

Thanks glad you liked it. Let me know if you have any questions.

[@Danieltammadge]

Have you started developing event driven systems?

[@thomas-sinkala]

@@Danieltammadge Yes, building my verify first now. Not sure if I am doing it right though ^_^

[@gurpartap_]

Hi Daniel! Thanks for the video. Question: Who decides which responses “didn’t make it earlier” and need to be handled by the notifier service? I’m sure one design won’t fit all, but any example with your way of explanation would unlock my neurons.

[@Danieltammadge]

It isn't so much as which response didn't make it earlier in this example. But more of email the person in case they are no longer waiting. What I have done in the past to keep things simple, is on success email the person who completed the registration or order etc. Why did I show it in a different service? Well you might have issues in sending emails, and then you would want to be able to retry and process later (kzbin.info/www/bejne/fYWrkomric2dlZI). Now that’s not to say you could not implement something a bit dynamic. For example, you could use web sockets and have the client open a connection with the notify service. And when the notify service consumes a message you could check if the client connection is still open and if still open send the result to them, but if the connection is not open then email instead.

[@mattyzacharia]

Well explained

[@Danieltammadge]

I am glad you found it helpful and taken the time to comment

[@khuiification]

Hi Daniel! Great video. From a cost-perspective id like to stay serverless and scale to zero so using WebSockets (and e.g. a cache to keep channel ids) is out of the question to me. Is there a way we can handle the asynchronous part in e.g. a BFF and still make it fit in a traditional reqyest-response model? E.g. The user still does a POST /register and the request stays pending util e.g. the UserRegistered event for that specific user appeared. How would you go about implementing this? As for WebSockets, i am curious how this would look in practice (even if it might not be what im looking for now) - do you have any example or are you able to explain how this would be done? I know it's a lot to ask but i want the architectural benefints with loose coupling and all of EDA, but i still want a simple request-response model which is what most people are used to and which doesnt add new complexities (both frontend and backdend wise)

[@Danieltammadge]

Hi Oscar, first off thank you for taking the time to watch and ask your question. When you say a cache is out of the question because you want to stay server-less and you want to scale to zero… what are you doing for data storage? If you didn’t want to use sockets then you can implement polling, and have the client poll for the response after a acknowledgement is returned back, but you will still need to have a data store to keep track of the status of the requests.

[@akhalil3482]

i was asked for event driven architecture at an interview, how can the client get data on load? i wasnt sure as client would need to send a request but that would be using rest? upon researching, it seems like having a websocket connection was one of the ways you can request data on load for an event driven architecture?

[@Danieltammadge]

Why can’t you have a REST API to fetch data on load? Who said you can’t mix the two? In solutions I’ve designed we have used REST APIs for fetching and triggering. But have used web sockets for the backend to push events to the clients. Also I would have a look at GraphQL APIs as GraphQL supports pub/sub.

[@ebrahimmansur9815]

good video .....small question how to deal with race condition like when making an order for 1000 gift cards (itune for example) and locking them

[@Danieltammadge]

Thank you for your comment Ebrahim. It is a small question. But the answer is not small. I have been writing out the possible solutions but a comment would not provide enough detail or clarity and won't answer your question in a manner I feel would suffice. I would like to reply with a video if you wouldn't mind. Please watch this space 😊

[@ebrahimmansur9815]

@@Danieltammadge I will be waiting....and again thanks for your great videos...keep going ✊

[@shubitoxX]

The timeout of request driven bringing timely feedback can be easily added to the event driven approach (listen for completion event with a timeout), I don't see a big difference between the two here.

[@Danieltammadge]

In request-driven, if the processor of the request is unavailable, the caller would know straight away and be able to gracefully degrade and log that it cannot perform the operation. However, once you put in place middleware, a queue or an event stream, and start using asynchronous event-driven communication. The publisher of the events is no longer coupled with the processor, the event consumer. The publisher won’t know if the processor is available or is not keeping up with throughput (consumer lag) because it will be able to publish events to the middleware regardless. An event processor is not time-dependent. It will process events as and when it consumes them. We usually hope it is as quickly as possible, but if you have consumer lag or problems, this could be minutes or hours. And when we introduce event-driven architecture into systems, we have to design for this additional complexity.

[@Danieltammadge]

Hopefully, sharing a walkthrough of one of my solutions will help. Users were unhappy with an existing service because the service was slow and unable to accept the number of requests in busy periods. And I was tasked with fixing it. However, we were unable at the time to improve the underlying issue, so we decided to change the service from synchronous to asynchronous and add a message queue to decouple the client and the processor so that the limit of the processor Would not affect the client's ability to send requests. However, not only did we have to change the UX and UI, but we also had to introduce the following: - A new version of the REST endpoint, along with backend components, to queue and log the requests. - A new REST endpoint for clients to fetch the status and results if required. - A Web socket API so connected clients can have processing events pushed to them. As well as the operations to serve on connect and disconnect operations and the extra database table to connect and store the connections for later retrieval by the backend. However, we also knew that due to the time independence processing of event-driven systems, some requests would take longer to process than the time the user would stay on the page which provided the service. Hence there was now a new requirement which didn’t exist when the service was synchronous, as we also needed to notify users who disconnected. So we also had to develop the ability to notify offline users via email. To conclude is that event-driven brings design considerations and additional complexity which don’t exist in synchronous request-driven solutions. Hopefully, this helps.

[@AndrewStarostin]

Thank you for the video! But how should Event-driven system behave if the registration was unsuccessful, e.g. username already taken? Should it publish the error event to the response queue for client to handle? Doesn't it just increase complexity by building some sort of an RPC on top of events without much benefit in this case?

[@Danieltammadge]

For your first question, how would you handle a failed validation? By all means, publish an event for the client to react to. You could publish successful and failed registration events to the same stream/topic/queue or publish success and failed events to different streams. You would have the outcome processor/consumer/subscriber processing the outcomes and pushing them out to the client. Be mindful of using publishing to different streams/topics and having consumers needing to consume more than one because processing events out of sequence can cause adverse outcomes. You may be asking. How could you push the result to the client? Will I see two approaches? The first is using sockets where the system keeps track of open connections and you pass the connection identifier in all triggered events. And you keep state of open connections, thus allowing you to push the results to the clients. Or you enable polling and keep state, and when registration outcomes get processed/consumed, you Reconciliate the outcome with the stored state, and the client polls the state to see if it changes over time. You should, of course, implement logic on the client for when the process is taking too long so that the clients are not waiting forever because a processor is not processing.

[@Danieltammadge]

As to adding complexity, m not sure what you are asking. Could you expand? Hopefully, my first reply answers your questions, though.

[@AndrewStarostin]

@@Danieltammadge Thank you a lot for the very detailed explanation, much appreciated! To detail my second question. We're sending and event to the server and waiting for the response to this event in the queue on the client. And client implements some kind of a timeout on top of it. And that is very similar to just using a sync request-response API, but requires more complexity to build. The only benefit I can see is that client can reconnect to the server and still receive the response event. So is it worth it in the case of user registration in your opinion?

[@Danieltammadge]

@@AndrewStarostin Is it worth it? Well, that depends on certain factors. One of those factors is the number of remote operation calls required to process a request. When an operation is processed by only one service or function, then it is properly not required However, if remote calls and distributed transactions are required to process a registration request, you will need to handle partial failures. But what happens if the service cannot finish processing and operations need cleaning up. You now have partially failed transactions that will need to be cleaned up. How will you handle this? In the request-driven world, you could be required to build a housekeeping process. And with remote calls in the request-driven world, if the services are down, you'll need to handle that, which will have an upstream effect, especially as requests are time-sensitive. As mentioned in the video, event-driven architecture improves retryability and fault tolerance and reduces the impact to upstream services when services downstream are unresponsive Another factor is; depending on how well the system handles processing load, you may need to throttle and apply backpressure, so queuing requests and returning a queue  acknowledgment allows clients to do something else, instead of keeping a connection open and still having to handle time outs and retires (you still need to handle timeouts and retries in the request-driven solutions). I would go with request-driven architecture if I had a single service or a monolith system that handles it all in one single transaction, and the time to process was low. Even in peak time, clients could retry serval times and have a high probability that a call would succeed. But as soon as you need to start working with multiple remote operations to process, then that is when I start approaching solutions with EDA.

[@AndrewStarostin]

@@Danieltammadge Amazing explanation! This is very helpful, thanks a lot 👍

[@harjisss]

Hi ! Thanks for the video. I was wondering what does Registration Client refer to? In web development world one could make a distinction between client-side and server-side and client in this context would mean end users browser. Is it so that in this case Registration Client is a server-side service?

[@Danieltammadge]

In this context it is more of a client side application which could be using web sockets to communicate. I did not want to go into too much detail in this video as I wanted to cover the two at a high level.

[@codewithkashif]

It is often said that having message broker help in asynchronous communication while in Request-Driven Architecture - api calls are synchronous and it need to keep waiting until response comes from server. So I have a question that C# "async Task" apis aren't asynchronous apis? Isn't that solving that synchronous api call problem

[@Danieltammadge]

Kashmir Thank you for taking the time to watch and ask your question

[@Danieltammadge]

No, it is not. And let me explain. Let's take a real-life example of a restaurant. Customers need to go to the counter to get served. In the first example, the server has not been instructed to be async by their manager. When a customer goes to the counter, they need to wait for a server to be available, and if they wait for too long, they will leave. When a customer is next, they give their order to the server. The server will serve the customer and only serve that customer until that transaction is completed. However, now the manager tells the server that while serving customers, they should try to serve other customers while they wait for another customer's order to be fulfilled. The manager has now given the server the async keyword. Now the server can use the time they would have spent waiting to complete the customer's order to serve others now. In doing this, the restaurant can serve more customers in parallel. Now that the server has now been instructed to be async, the server can now use the empty time to perform other activities. Doing so allows the restaurant to serve more customers in less time. This is because the customers waiting time to get to be served is decreased. However, the underlying type of communication between the customer and the server has not changed with or without the async instruction. Thus the same problems still exist. For example, if the server needs to leave the counter, no orders will be taken, and customers may see the restaurant has no servers and leave. Also, if it takes a long time for the customer to give their order, the customer may become unhappy and leave. And the customer is tightly coupled to the server; the customer cannot do other activities apart from waiting at the counter. The async instruction to the server now makes the server more effective in using their time, thus having positive impacts. But doesn't change the underlying communication between the customer and the server. This above approach is a synchronous request (point to point) Now let's say the manager sees that he can increase the number of customers he can serve, and it involves removing the need for a customer to have to give the order to the server directly at the counter. So the manager hires order takers as well and sets up an order station and a pick-up station for the customer. The customer still needs to go to the counter to order, but they go to the pick-up point to wait for their order once after they order. The restaurant can now take customers more orders in the same amount of time. But the customer still needs to perform the work to get their order. They need to keep looking at their order status to know when to get their order. This is an asynchronous request API that uses polling. The manager sees that even though the counter can now take orders quicker, customers still need to wait to collect their orders. They cannot go back to their tables and do other activities until they have their order. The manager sees that this is causing customer dissatisfaction, so he now has employees take the orders once ready to the customer table. This allows the customer to return to their table and enjoy their time once they give their order. This is now asynchronous request APIs using webhooks Customers are happier, but the manager still thinks improvements can be made because, at the moment, the customer still needs to get up and talk to an order taker (or wait for the order taker to go to their table). This is still a problem because the ability to get orders is limited by how quick the order takers are and if they are available. The manager sets up an order taking solution where customers can now order via an app. And the bar receives the orders via an order system. The customer can order items without direct communication with an order taker. They can order anytime with their app, no waiting. And the restaurant no longer needs order takers and can now focus on fulfilling orders. The manager is happy he can now take more orders and fully focus on making the orders. Still, the manager knows that to keep customers happy, he needs to ensure that the restaurant's ability to fulfil orders does keep up with the order throughput and does not lag. And even if the bar's order screen goes down and the bar doesn't receive orders, the customers can still make orders. And when the screen is rebooted, or the problem is resolved (hopefully quickly), the bar will not have lost orders and can continue fulfilling the orders made. The bar is now using an event-driven approach with a broker (or platform), which enables in-direct loosely coupled communication using a broker (or a platform)

[@Danieltammadge]

Would have liked to reply earlier to your question but I wanted to provide a deeper insight. I hope it helps

[@codewithkashif]

@@Danieltammadge Awesome metaphor you have taken here! Thanks a ton for putting so much detail in it! Thank you for taking the time to help me, it is really really very helpful.

[@Danieltammadge]

Happy to help. And I’m so glad you found it useful

[@n3x404]

pretty cool video gg

[@Danieltammadge]

Thanks N3X

[@luckys9310]

Do u have a udemy course?

[@Danieltammadge]

No I do not. Everything I have published so far is free on KZbin or my blog

[@ikechimike6894]

Please include subtitle I'm h-impaired Please

[@Danieltammadge]

I’ll be checking my videos soon to ensure they are have subtitles

[@ikechimike6894]

@@Danieltammadge thanks