Unlock the Power of Event-Driven Architecture: How Netflix & Uber Handle Billions of Events

[@danser_theplayer01]

With this need for efficiency web development architecture becomes more and more reminiscent of hardware architecture. This idea of just throwing any signals out there and expecting every different service to handle or pass them differently on their own reminds me of something like cpu architecture.

[@ByteMonk]

That's an interesting analogy! You're spot on-modern web development architecture, especially with event-driven systems, is starting to mirror how hardware architectures, like CPUs, handle signals.

[@aethermass]

Good video. For those wanting to dive deeper, the Akka Actor library and the actor model are good topics to read up on.

[@ByteMonk]

Previously I made a video on Paypal's use of Akka here. You might like it too :) kzbin.info/www/bejne/a4PdmYaEZtprers

[@jordyvu999]

I once researched akka, not sure why it not popular use now?

[@aethermass]

@@jordyvu999 Most people/companies don’t need to build systems that require it. So it is specialized knowledge. Additionally, it has a lot of functionality and licensing fees. For instance, Apache Spark used it for a while and implemented the pieces they needed once figured out exactly what they needed. The actor model is super useful. Akka is just one implementation.

[@avwie132]

@@jordyvu999scala is not that popular anymore. Elixir is also very interesting from an actor perspective.

[@RaeWar]

@@jordyvu999Licence, better use Apache pekko

[@tarunstv796]

I have following question, please bear with me for the basic/stupid ones - 1. How to decide on EDA vs Syncronous communication? 2. When a single event can trigger multiple services and they react to it differently then how their reactions are captured? For example - uber example - for ride request - price calculation service, matching service etc. will do their respective jobs but 'who' and 'how' will aggregate all this for uber's customer? 3. Event Processing: Simple and Complex - How are multiple events aggregated as they can arrive at different times and can change (event1: high demand, event2: rain, event3: low demand, event4: sunny weather)

[@littledidsheknow]

1. It’s mostly based on how long you take to process a request. If the processing takes longer better to go with async and respond back immediately with a 202.

[@ByteMonk]

Of course! And don’t worry-it’s totally normal to feel a bit lost with these concepts at first. When choosing between EDA and synchronous communication, think about how independently services can operate. For example, in Uber, EDA works well since pricing, matching, and notifications can each respond to a ride request without waiting on each other. In contrast, synchronous communication is better when services need quick, coordinated responses. For handling multiple services that react differently to the same event, an orchestrator service can gather each service’s response and present a unified result to the user, like Uber showing fare, ETA, and notifications together. And for complex events (like high demand plus rain), Complex Event Processing (CEP) helps by capturing patterns across multiple events and triggering actions in real time. Do checkout all videos in my microservices playlist where I talk about each of the patterns and hopefully you will be a pro in no time ;)

[@mahendragundeti]

We use all the patterns you tell I can connect all dots thanks 🙏🏻

[@rakeshagarwal9150]

Excellent information 💯💯😊

[@ByteMonk]

Thanks 😊

[@parashar1505]

wow! what software do you use to make these graphics? it is just so great!

[@akhilrafftar6011]

I love the way you present your knowledge in such a simple words with good visualisation

[@ByteMonk]

Thank you so much for the feedback, simplifying complex topics is always my goal!

[@GauravPal-pd9rf]

Nicely elaborated, appreciate your efforts.

[@sahilq312]

Great explanation .

[@rakeshranjan5071]

Amazing explanation. And great animation

[@bensdevjourney7081]

What animation software do you use? Great video 👍

[@manojmaximum]

top notch content, animation, voice

[@julienwickramatunga7338]

Nicely explained, thank you!

[@aravindd4911]

KEDA 😍

[@deepanshuchoudhary4598]

how do you generate those animations/diagrams

[@RupeshBhandari-977]

Following

[@H0nGzA1]

i want know to

[@kgothatsontsane3119]

.

[@AdaptorLive]

Generate makes it sound like it's automated. I think these animations are all custom made.

[@texhio]

I guess motion canvas

[@mitesh5189]

Amazing information thanks. one suggestion could please add some open source project is also using such patters not in just this case but in general it will be beneficial thanks.

[@yurhighness1998]

awesome men

[@muhammedadel9673]

NATS is also one of those new event distribution channels that's being adopted recently in Netflix and it is open source

[@Arun_Poulose]

Which software are you using to create just diagram drawings?

[@ЦзинКэ-ы5х]

I think you should've also mentioned that any element in the system (depending on how the MQ is handled, what broker is used, etc, but still) can be both consumer and producer. By the way, aren't "publisher" and "subscriber" equal to producer and consumer at 6:58?

[@josephrana-np5bb]

Nice content

[@AlternativPerspectiv]

What did you use to create the graphics?

[@shubhamdas6519]

Thanks a lot sir

[@softwareengineer8923]

Such a useful content, thanks a lot

[@ByteMonk]

Appreciate the support, happy you enjoyed it :)

[@ALEXANDER3TON]

how they handle network flow like the video?

[@oatlylatte9]

so is EDA is the best architecture so far that could accomodate our needs?

[@ByteMonk]

It depends on your specific needs! Event-Driven Architecture (EDA) is great for systems that require decoupling, scalability, and real-time processing. It works well when services need to react to changes or events asynchronously. However, it might not be the best fit if you need strict consistency or if your use case doesn't require the complexity of event handling.

[@Someonner]

Error here SQS is not used in Lieu of Kakfa but Kinesis is used. SQS is messaging queue whereas Kafka is Streaming Queue

[@nikhilkedia2252]

Which software was used for the animation.?

[@drblahblah929]

Great content with great presentation. I really like how you made this video. Can please tell me which tools you used to make this video?

[@wwalsh0351]

What are you making these diagrams with? Sorry for the sort of unrelated question.

[@codertravel99]

Amazing explanation but if possible then try to implement all these patterns using microservices so that we can see realtime usecases that how we can use them in realtime applications like uber and netflix and etc

[@developer00007]

New subscriber added

[@princeraheja8737]

Need Synchronous architecture video

[@ByteMonk]

Noted :)

[@satish1012]

Great presentation You mentioned SQS is one of the EDA But , the problem is it is Queue based . Only 1 consumer can get it the message. I think AWS event Bridge or SNS or kinesis s the right one for so that each consumer gets its copy

[@ByteMonk]

You're absolutely right! SQS is indeed a queue-based system where typically only one consumer processes each message, which might not fully align with a true event-driven architecture where multiple consumers need their own copy of the message. For that, AWS EventBridge, SNS, or Kinesis are better choices since they allow multiple consumers to receive and process the same event. Thanks for pointing that out, and I appreciate the clarification! Will make a follow up video on this topic in future.

[@Clma333]

This was such an insightful video! I really enjoyed the part about Netflix . It reminded me of a video I recently made on VPC, where I dive deeper into core concept and implementation of VPC. If anyone’s interested, feel free to check it out - I’d love to hear your thoughts!

[@cybergaz]

how did machines know that an event is occured?? if your answer is by continuously checking for incoming events, than how is it different from request response architecture??

[@priyansh1210]

Found this on Quora: Sometimes it’s polling/busy-waiting (checking over and over), sometimes indeed the CPU fires off instructions (interrupt request). For example if you’re writing server code in NodeJS, on a supported operating system with async I/O, it’s driven by interrupt requests from the network card, no polling required. The network card, when it receives data, will send certain voltages on certain pins, that will cause the CPU to jump to a specific piece of code in the drivers and the OS, and from there to NodeJS, and then finally to your event handler in JavaScript. That’s why this is so fast and efficient. P.S. NodeJS will actually be “checking over and over”, except on each check, if nothing is found in terms of events waiting to be handled, it actually gets frozen, and only unfreezes when there’s an interrupt request, either by timers or by hardware. So it’s not wasting CPU cycles when there’s no work to be done. P.P.S. True busy-waiting, or “spin locks”, are also used in practice, but only in special cases, where it’s expected the event will happen after an extremely short duration of time, where it’s just not worth it to suspend and then wait for the interrupt handler routines and stuff. But when I say short, think nanoseconds, not milliseconds.

[@BradleyWeston92]

pubsub has a lot less overhead and can implement retries etc and is async

[@BradleyWeston92]

plus you do not need to know who consumes your events, they just ask the messaging provider

[@cyberguroo]

These machines are "subscribers" to those respective events. It's the message brokers ( eg. Kafka) that makes a call to this subscriber that there is a new event in the queue that you are subbing to. Otherwise these consumers sit idle. They don't continuously poll for new events.

[@ByteMonk]

In a true EDA, machines don’t constantly check for events. Instead, they rely on event listeners or subscribers to be notified only when an event actually occurs. This is different from polling (where you constantly check) because, with EDA, the system uses push notifications rather than pulling for updates. For example, when an event occurs (like a message arriving or data being updated), it’s pushed to the listeners through an event broker (like Kafka or RabbitMQ). These listeners or consumers only act when they receive a new event-no constant checking is needed. This makes EDA efficient and scalable, unlike request-response, where each interaction requires a direct request to get information. So, rather than polling, EDA uses a publish-subscribe model to handle events only as they come, reducing unnecessary load on the system.

[@malliktest9160]

Great content. How did you draw the diagrams. They are so pretty, I am planning to incorporate samething in my presentations

[@dushyanthchowdary1388]

So, these services are microservices?

[@BradleyWeston92]

doesn't have to be, monoliths can use events, but essentially if you use "micro services" you want to ensure reliability by using an event driven arch

[@jordanstewart225]

I disagree with the premise. All your services don't talk to every other service. You normally have a dependency on only a few services, if that.

[@ByteMonk]

Thanks for your input! You're right-most services in a microservices architecture usually depend on only a few other services, not all of them. The video’s focus was on the potential challenges of inter-service communication in more complex systems. That said, in practice, many services do have minimal dependencies. I appreciate the feedback, and it's an important point I must clarify in future videos!

[@jordanstewart225]

@ByteMonk I think event driven architecture seems to stand out more with data processing, like with ksql and ktables with Kafka for example

[@SDAravind]

Can we use redis streams?

[@znpz]

Avoid both, ideas, code, thieves from interview.

[@jay_wright_thats_right]

Please use an AI voiceover. Your accent is too thick and I can barely understand your English.

[@alexm6762]

Unimpossible to listen to this video due to the terrible and fast accent like single noise

[@ByteMonk]

Thanks for the direct feedback, I have received mixed opinions on my pacing and articulation/accent. I will keep working on them. Meanwhile, there is a 0.75x speed option which you might like to use.

[@arazDev0]

try explaining what EDA is... the video lost 90% of its value because you failed to explain to me what it is that you are talking about. i hope this feedback helps you make better videos in the future.

[@ByteMonk]

I appreciate your feedback! I did my best to explain EDA (from 1:16 to 1:49 ) in the video, but I understand that sometimes things might not come across as clearly as intended, and it probably depends on the level of experience. You might like to checkout my microservices playlist. Thanks for watching and for your honest input-it helps me improve!

[@أيوب-خ4ب]

Tbh, I was struggling understanding what EDA is and watched many video and read a lot of post and I couldn't not understand it. I came across your MapReduce video a few days and it helped me understand how it works, so I thought you might have an EDA video and you did. And now I CLEARLY understand what EDA is and how it works. The diagrams and your explanation is spot on and helped understand how these things work Good job and keep up the good videos

[@user-mu7tw2mt8k]

@arazDev0, he explained as clear as day what EDA is. You should watch again, and don’t be too quick to discredit someone else’s work. He never said it’s a pre-beginner level video or a very comprehensive one, but it summarizes the general concept well. May be going over the very the basics and terminologies first will help.

[@ASHWANISHARMA-ng9ix]

If i use socket io for api that also be a eda 🙄

[@BradleyWeston92]

No. Events are published to a message broker, this means the service doesn't need to know about who is consuming it. The consumer of the events can be 0 to X amount of services and makes no different to the publishing service. The broker will also manage retries and other semantics.

[@ByteMonk]

In EDA, services are generally loosely coupled and communicate asynchronously by producing and consuming events through an event broker or message queue (like Kafka or RabbitMQ). Each service listens for specific events and only reacts to those it’s interested in, which makes scaling and evolving the system easier. Socket.IO, on the other hand, is great for real-time, bidirectional communication between a client and server-often for applications like chat or live notifications. But it’s typically not used to decouple services or handle asynchronous, long-lived events across multiple services. So, while Socket.IO can handle events, it doesn’t provide the loose coupling and scalability that a true EDA setup would offer.

[@whitespace-jg5jj]

I love the visual... but the accent is frustrating. Like I have to debug it too so I could understand every single word you say.

[@AfiqSazlan]

If you’re frustrated with someone’s accent, make your own video. He/she doesn’t owe you anything.

[@ByteMonk]

can you please point out specific parts of the video or words you had difficulty understanding? I can work to neutralize my accent. You may also use youtube subtitles, I noticed it picked up everything properly :). Thanks!

[@Theonewhoescapedfromcommie]

Architects are weird. They first tell us - normal happy hard-working developers - to build services and connect them by HTTP protocol under RESTful label and call it "microservices". They are so proud of that, it's like they have invented "the light bulb" of the 21st century. Suddenly, they found the problem of "connection hell", then they introduce something called "event-driven" which is anti their own pattern that makes everything bottleneck again (make bottleneck be the problem at all time - MBBPAAT) to solve "connection hell", again they invented another "the light bulb of the 21st century". Hura hura

[@ByteMonk]

Haha, I totally get where you're coming from! It can definitely feel like architects are constantly shifting between "the next big thing." First, RESTful microservices seem like the perfect solution, then, boom-event-driven architecture is introduced to solve the same problems that microservices were supposed to fix. It’s like they’re inventing a new "light bulb" every time they hit a new challenge. But honestly, each of these approaches addresses different problems as systems grow in complexity. RESTful microservices were a great step forward in making services modular and scalable. However, when the number of services explodes, "connection hell" becomes real, and that’s where event-driven architectures come into play. It's all about trade-offs, and yeah, sometimes it feels like we’re just going in circles trying to find the "perfect" architecture that never really exists. Thanks for the laugh though-your comment hits on something many developers feel!

[@sathishd273]

Awesome. Very clear explaination. Thanks