How Microservices Communicate? Sync vs Async. Direct vs Brokers And Event Busses

[@DevOpsToolkit]

How did you organize communication between your applications? IMPORTANT: For reasons I do not comprehend (and Google support could not figure out), KZbin tends to delete comments that contain links. Please do not use them in your comments.

[@n4870s]

What about cross service transactions?

[@DevOpsToolkit]

@@n4870s Wait until next Monday. I'll publish a video that will go through one possible technical implementation on that subject.

[@emonymph6911]

Love the overview. Where is your followup video called for how to create Async driven events with a broker? I recommend adding it to the description of this one and maybe a link at the end of this video, thank you.

[@DevOpsToolkit]

That video has not yet been done. It's still ok n my to-do list.

[@fenarRH]

This is a good topic which I end up having conversations on, the way I open the conversation is to marking the direction of communication (North-South vs East-West) in to app-farm (ie microservices colony). Most of the time N-S comm is connection oriented (Direct/Sync) and E-W is async however what I see in legacy to app-modernization projects E-W also kept as Direct/Sync which is a perf killer also roadblock for scaling up the bottlenecked pods but endup scaling whole app-farm.

[@RishiRajxtrim]

I have been watching this video every year to under microservices and the communication among applications even better... every single time. Very powerful words delivered. Thanks a lot!

Event-driven architecture should not be the default choice because depending on your requirements, you might need to prioritise messaging over events. For instance, if replay-ability is not a concern, you do not have to opt for an event-driven architecture. Instead, you might value transient data more than historical data. Furthermore, persistency may not be a critical factor, as messaging allows consumers to acquire a message from the channel/queue, which subsequently gets deleted. However, in Kafka-like services, a retention period exists, enabling consumers to change the offset and re-read the events even after consumption. In such cases, there might not be a necessity to choose event-based communication.

[@davoodgoojposht]

Thanks a lot, It was a very helpful video. Only one thing on the topic of "Sync calls not being resilient", I think async calls have the same behavior too and even worst. With the Sync call if a service is down at least we get some feedback and error, with Async if one service is down we get no notion of a problem. so neither are resilient, we just don't get any errors in async

[@DevOpsToolkit]

If a service is down and if it starts processing messages in a queue, it will eventually catch up.

[@davoodgoojposht]

@@DevOpsToolkit you are right, but also in many occasions the cathing up is more than a problem than an advantage, users keep resending the same request because they are not getting any response, and we should clear the messages before bringing up the service. I personally believe the fast feedback of sync communication is actually a great advantage.

[@jimbrown6422]

Viktor, Really great video. You explained it in easy to understand language. Keep up the great work. Thanks for this. Jim

[@bjoernbung]

Thanks Viktor! Very good introduction. Looking forward for the next video.

[@iham1313]

very helpful differentiation of a/sync and de/coupling - thank you very much on that

[@KrSunilPatwa]

Hey Viktor .. Thanks a lot for explaining this topic in a simple and easy to understand way. Really loved the video .. I am basically a java Backend Developer and just subscribed your channel Hopefully will leran a lot from you and your Videos Once again Thanks

[@kevinyu9934]

Very inspiring!!! I like this topics. Super pumped for the next one,

[@dmitriinadein4733]

That is awesome! Please post a link for next video, if possible. Many thanks, we are watching your channel with collegues.

[@ayazahamed8254]

Very well explained. It help me to understand how we can communicate the microservices.👌

[@naresing]

Great !! Keep up the good work Viktor.

[@SV-tc8cu]

Excellent video, short but full of useful concepts

[@erikslorenz]

Viktor I love your channel. Thanks for the content!

[@MrTouqeerhameed]

Great video -This is a thing that i am looking for

[@НиколайЗаднепровский]

Good quality of content, thanks!

[@geowalrus]

That communication choreography is sometimes called 'saga' in event driven architecture.

[@denisstortidasilva3832]

Great video! Excited to see more videos around architecture and integration. I like the technical videos too, but I've seen top edge tech adopted with poor architecture principles and it doesn't end well :) thanks!

[@DevOpsToolkit]

In this video, I'm experimenting with a new format. Typically, I'd explain architecture/process/etc. first, and then go through the technical parts and a demo. This time, I thought to split those so the next one will be one possible implementation of the ideas presented in this video. I'm still not sure whether having it together or split makes more sense so I'll do both approaches and see the reactions before I choose to stick with one or the other.

[@denisstortidasilva3832]

@@DevOpsToolkit splitting is a great idea, as you can explore different approaches/technologies to solve the same pattern (eg. Async integration by using threads/workers at application level, versus async event driven messages + choreography/orchestration) . I thought this one could be even longer, as the concepts are not easy to explain. Looking forward to it! 😍

[@DevOpsToolkit]

@@denisstortidasilva3832 Such feedback is very helpful in deciding how to organize future material. Thanks. As for length... It's very hard to decide long long videos should be. When I make 30+ min videos, some people say they're too long. When I make them shorter, some say that they'd like them to be longer. That's also one of the things I want to experiment with. I'll try creating videos with more varied length (from a few minutes up to 45 min.) and see what has better reception.

[@erikslorenz]

@@DevOpsToolkit I like the new content. I work in a smaller company and I think that a lot of people in a smaller company may not think this is valuable. But I spend a ton of time integrating services and having good architecture between services is incredibly helpful. I use a lot of Google pub sub. As soon as you find yourself with 6 or 7 or 12 applications you need to take a step back and evaluate.

[@DevOpsToolkit]

@@erikslorenz It's normal for smaller companies not to think about practices that are designed for bigger operations. The problems typically occur when something small becomes bigger and teams do not take time to "step back and evaluate" the new situation but just keep going.

[@WilkoWenzel]

Danke!

[@DevOpsToolkit]

Thanks a ton!

[@amitsahuit]

Wonderfully Explained.. You are awesome.

[@opelfahrer91]

Really liked that video, thank you :)

[@akitmentorconsultant4696]

Great video 👍 the same thoughts in my head too. Actually, synchronous communication is not a natural / ecological way to communicate. Because, real world works asynchronously. Synchronous approach is only problem of HTTP1 from my point of view. HTTP2 a little bit better see Server-Sent Events. But how many time/money we spent with HTTP1.1? A lot of years)

[@dmytroshchotkin2939]

This was a brilliant video!

[@Fayaz-Rehman]

Better than ever. Thank you for sharing.

[@SoftwareDevelopment-zy1wh]

Nice Explanation. Thanks

[@chikosan99]

Thanks a lot!💥 You have great content

[@iaroslavdavydiak6439]

Awesome 🚀👍, as always🙂. I like your visualizations, it helps me to understand what you mean.

[@DevOpsToolkit]

Thanks a lot for the feedback. Visualizations are new and, so far, we used them in only a couple of videos. I'll double down on them and use them more if they're helpful.

[@geraaksenov4688]

Thank you for the great video! I have a question: in asynchronous communication, how can we prevent broadcasted event/message to be processed multiple times by different independent listeners? Because if we don't, events may be multiplied (e.g. event of putting a product in the basket will be picked by multiple processors and each of them will order this product from the storage).

[@DevOpsToolkit]

Typically, events should be idempotent so that multiple handlings of the same event do not result in multiple records. Otherwise, it's hard to guarantee that events will be processed in the first place. That's why we normally adopt the at-least-once approach. That being said, you can set up the backend of the event to guarantee that an event will be picked up by a listener only once.

[@geraaksenov4688]

@@DevOpsToolkit By "backend of the event" do you mean the message broker?

[@DevOpsToolkit]

You're right. I meant message broker. I should stop writing messages over my phone while crossing a street :(

[@geraaksenov4688]

@@DevOpsToolkit I appreciate your effort :) Thank you for the answers

[@camaycama7479]

What can act as a broker in kubernetes? Say I have a dozen cluster in k3s, what service I can use as a broker?

[@DevOpsToolkit]

Try NATS.

[@devopscraftsmanship9302]

Great content! I came across this channel this year. I wish I did earlier ❤

[@MrR3set]

i want to do an admin service that will manage all admin actions across my apps. ADM will manage role authorization and admin logging (audit log). I was planning on using direct communication between services. Each service will have their own admin endpoints exposed just to the adm service. Do you still think eventd are better here?

[@DevOpsToolkit]

I can't answer that question without more info. What i can say is that direct communication is easier while events are more reliable abd scale better.

[@willnguyendev]

How about a sync rest request need the result of second service? first service needs to send message directly to second service and wait for response, doesn't it?

[@DevOpsToolkit]

It does, but not in the mode I used it. I used pub/sub while you want direct messaging.

[@jaturontepjuk3046]

Nice background color, mate

[@DevOpsToolkit]

Thanks

[@TechStory5]

Can't wait for the next video !

[@DevOpsToolkit]

It's coming next Monday :)

[@javisartdesign]

thanks! great explanation this is a big such topic

[@susanasher6939]

Does the broker send the message out only to the applications that are supposed to hear the message, and one of the ones that hear the message responds and the one that responds first gets to do the task? Or, does the broker send out the message only to the applications that can do that certain thing that is being requested?

[@DevOpsToolkit]

It all depends on how you configure the broker. Also, more often than not, they don't send messages but allow other processes to subscribe to messages of certain type/subject.

[@susanasher6939]

@@DevOpsToolkit Thank you so much for answering and making it easy to understand this!

[@arto00-g2n]

Great video thanks

[@ptdev6776]

So basically service mesh tools like Istio that are based on synch communication...are bad tools? Which should be the sense of these tools if synch communication is a bad thing?

[@DevOpsToolkit]

Sync communication is unavoidable. For example, a backend app needs to talk directly to the db.

[@spottedmahn]

What about, I want to GET ____? Post an event? That doesn’t seem right. What am I missing? 🤔

[@DevOpsToolkit]

If you want to get something, that's a simple get from the service or, more often ,a database that has the info. Events are more about "i just posted this, whomever needs it, come and get it." Imagine it's Amazon that consists of many services and someone just added an item to the shopping cart. That specific service does not need to worry who else needs that info. It does not need to do anything but post an event knowing that whomever needs to do something with it will get it. It's about decoupling. Another example would be kubernetes itself. When you send a manifest to Kube API, an event is created. The API has no idea which controller is responsible to do something with it. Instead, controllers are listening to events and performing actions when it's one they are subscribed to. Otherwise, the API would have to have the logic of forwarding requests to a massive number of controllers or you, the user, would need to know which specific controller to talk to.

[@muratasarslan2359]

Very well explained, thanks 😃 What would you think using gateway when we need sync communication?

[@DevOpsToolkit]

If you need sync communication, any service mesh should do.

[@muratasarslan2359]

@@DevOpsToolkit what if the microservices are subgraphs in an Apollo Graphql Federation 2 ?

[@DevOpsToolkit]

unfortunately, I haven't used Apollo 😔

[@puneetsaneja3007]

How Synchronous microservices talk to each other on cloud i.e on AWS without using QUEUE. Suppose there are 2 microservices A& B which are deployed on AWS ec2 instance. I cant hardcode B endpoint inside A microservice. Bcz everytime Ec2 instance started , the IP gets changed.

[@DevOpsToolkit]

In that case you would probably talk to those services through a fixed endpoint like ELB or ALB.

[@puneetsaneja3007]

@@DevOpsToolkit so u mean, instead of calling B microservice directly , we call load balancer which redirect our request to B. Right ?

[@DevOpsToolkit]

Yes.

[@mohamedaittaleb]

Great and easy explaination, Thank you, waiting for your video on Kafka ;) ---- am I the only one who use the Playback speed 1.5 xD ?

[@DevOpsToolkit]

You're not the only one that watches with increased playback speed. I do not think Darin ever watched anything at 1x :)

[@soubinan]

Yeaaahhhhhh!!! Waiting for the next :)

[@DevOpsToolkit]

It's already recorded and is currently going through the editing and animations phase. It'll be published next Monday :)

[@BPTScalpel]

I'm really confused about why you think sync is not performant and how exatcly it helps in your example at ~8 min. App1 is waiting for a response. How do you improve the latency of this response with async? If you truly care about performance in this example you actually make everything sync with long lived connection, don't you?

[@DevOpsToolkit]

If you look at the performance of that request, you are right. But, if you look at the performance of the application as a whole or the whole system (especially at scale), waiting for responses is costly, especially if that means requests to multiple applications.

[@saynay302]

You can think of it like the difference between response time and throughput, especially under load. Sync calls are great, until they hit some threshold where all their threads are blocked waiting for an operation to complete, then they tend to explode. Synchronous operations are fine if you have a high certainty that they will be completed quickly. The issues can come if you have a chain of sync calls, as any point along that chain could get stuck / crash / etc. and force your sync calls to block until they time out. They also don't handle changing scale gracefully; if you have a number of requests in-flight and the system decides to scale-up the service you are talking to, those in-flight requests will not be able to take advantage of the new capacity. If those requests were instead sitting in a message queue, when new processing power came online they can be handled right away. Async has some tradeoffs though. It is generally more difficult to reason about. You often need to handle scenarios you do not in sync (like retransmits, or duplicate messages), and will generally not have the same guarantees around ordering.

[@BPTScalpel]

@@DevOpsToolkit Yes, but you would still have to wait for the response even in an async architecture; it would just not be the responsibility of App1 but of a different consumer somewhere down the line. The fact that you need the response to continue does not disappear because you change protocols. Don't get me wrong: I love async for fire and forget scenarios, but saying sync is not performant by taking a RPC call as an example (what it is especially good at) is a bit weird. Now, if what you are saying is that you need more tools to make sync work for RPC at scale, I agree. You would need both clients and servers to be able to handle requests asynchronously, you would need to add a load balancer, retry mechanisms etc... all of which come out of the box with async but nothing can be more performant (i.e. provide a faster response) than sync for RPC in my opinion.

[@DevOpsToolkit]

@@BPTScalpel You're right. I made a mistake by not explaining it well. A direct sync reqest/response is performant.

[@emonymph6911]

I'm confused what a broker actually is, is it a database or a deployment service like Kubernetes or service like Dapr? Please make an independent video on brokers, and how to connect to them and make them communicate with other third party applications or APIs.

[@zohaibsaeed4063]

Example Hive mq for mqtt.

[@HosseinMehradbahiraei]

Thanks a lot

[@AnkiteshSrivastava]

I don't understand, why you still haven't reached 100k mark

[@DevOpsToolkit]

I guess that not doing any marketing and not focusing on newbies leads to much smaller number of subs. I'm working on this channel mostly because I enjoy it and certainly not as a business so I'm not doing what needs to be done to increase subs.

[@DaljeetSingh1]

you are awesome !

[@bileldridi324]

Really interesting topic ! if we use such a communication system, we can reach easily an important complexity in big projects. So how can we get the big picture of microservices communication ? Do you know some documentations tips or tools that can solve tis kind of problem ?

[@DevOpsToolkit]

Tracing tools like Jaeger can help with that.

[@YourTechBudCodes]

Amazing video! Loved your take on how microservices communicate. I've made a playlist on this topic which I would love for you to checkout. I'd like to add one thing. You mentioned that HTTP is a synchronous protocol. While that's completely true, you can still use it for triggering async events. Similarly you can use AMQP to implement synchronous RPC calls. So the nature of protocols doesn't directly affect how your microservices communicate.

[@DevOpsToolkit]

I think I said that a couple of times. The fact that one protocol is synchronous does not mean that it cannot be used asynchronously, and vice versa. Now, it's quite possible that even though I said that, I did not explain it well. I'll do my best to improve the way I explain in the upcoming videos.

[@razbuchnik6007]

Didn't answer the most important question: how to handle ETLs of duplicate data.

[@DevOpsToolkit]

You mean that you are trying to store data that is not uniquely identifiable so you might end up storing more than one copy?