Message Brokers - Introduction

#microservices #messagebrokers
Message Brokers are an Asynchronous Inter-Application Communication Technology. In this video Introduce various Message Brokers and concepts that you need to know and understand about Message Brokers in general while providing you with real world architectural examples for Why and When to use Message Brokers.
Chapter Markers
00:57 What are Message Brokers
01:20 Kinds of Asynchronous processing
01:22 Async Background Work
01:51 Async Fire & Forget
02:26 Async I/O
02:51 Out of Band Async
03:43 Why or When would you use Message Brokers
04:01 Send An Email Using MSMQ (Simple Architecture)
09:56 SendAn Email Blast Using MSMQ (Simple Architecture)
11:36 Send An Email Using RabbitMQ/ServiceBus/Kafka (Enterprise Level Architecture)
15:58 Various Message Brokers that are available
19:25 QueuesTopics (or Exchanges)
20:56 Queues Competing Subscribers
22:39 Topics (or Exchanges)
27:07 Archtecture of a KZbin like Application (CompetingSubscribers/Load Leveling)
33:19 Real World Scenario 1 Problem Description
34:42 Real World Scenario 1 Solution (Load Balancing)
39:12 Real World Scenario 2 Problem Description
40:33 Real World Scenario 2 Solution (Load Leveling)
45:08 Delivery Options

[@user-ij5sy5fc2i]

Great job, Awesome video!

[@nikhilramabhadra6052]

Gr8 video on message brokers. Additional async: 1) APCs in Windows 2) DPCs in Windows to handle interrupts 3) IRQs and IRQLs in Windows 4) Tasks in RTOSs 5) Task schedulers 6) Signals and slots in QT 7) Signals in Linux 8) Polling seen in embedded systems, Windows, and Linux. 9) Shared memory 10) RPCs

[@Matlus]

I wasn't listing out various async/interrupts but rather when folks typically say "async" it could have various meanings. and thus not very clear. For example, if I asked someone to implement something as async, it may not be clear what I really mean. Are you saying I should make the call async when talking to the database? Or do you mean I should spawn a thread and do call the database from within the thread? Out-of-band async means, don't do this work in this process.

[@nikhilramabhadra6052]

@@Matlus Please could you go into load balancing for an overwhelmed receiver. The database example you talked about in the video. I wanted to know on how the traffic to the database was handled without overwhelming it.

[@saurabhchauhan232]

As always superb explanation of Basic or history before getting started. Appreciate all your hard work you do.🙏

[@Matlus]

Thank you Saurabh!

[@shaikzuhair8537]

Iam eagerly waiting for this topic... Finally thanks for uploading this lecture

[@Matlus]

Most welcome!

[@harshitshah4158]

Great.. informative and very helpful to understand broker world.

[@Matlus]

Thank you Harshit! Glad you think so.

[@HemantKumar-yk2jk]

Thanks a lot.Awesome video...

[@Matlus]

Glad you liked it

[@SuperSynthguy]

what is this gift of a video!! simply incredible. A slow conversational gait is super helpful for soaking in high-level information, thank you for taking your time while speaking !

[@Matlus]

Thank you Paul! I'm glad you enjoyed this style and pace.

[@SuperSynthguy]

@@Matlus I love your real world examples and expertise. Amazing that we can find this type of info on KZbin these days. Im 25, stepping into more architectural development and design and these videos are helping add a lot of color to the space coming from a less experienced point of view

[@Matlus]

@@SuperSynthguy Glad you're enjoying these videos and learning from them. Keep learning and coding. All the best!

[@DotnetCoreTelugu05]

Shiv, your videos have great depth knowledge... If I want to understand, I need to repeat so many times. But It motivates me to learn new concepts.

[@Matlus]

Thank you! Yes, and what you understand when you watch these videos every year will be vastly different from what you understand today :).

[@DotnetCoreTelugu05]

@@Matlus By Experience I can..

[@yengobill3603]

I love @Shiv Kamur. Bust content I've ever consumed. His channel is amazing. He puts the meat on the bones on everything so to speak. I really wish I found him earlier.

[@Matlus]

Thank you Yengo! 🙏

[@amrita7497]

Great video 👌

[@Matlus]

Glad you enjoyed

[@srieen100]

As always great video Shiv, looking for the rest, can you please couple all these and show a grand picture for Microservice architecture

[@Matlus]

Thank you Sriram. I can try, how I don't know what you mean by by Microservices! :)

[@raviakella2877]

Nice one Shiv 👍

[@Matlus]

Thank you Ravi!

[@sumitmore4680]

Like always great video, Shiv!!! I am fascinated by messaging/event driven systems. Tnx for video! I am highly interested in ordering of messages as I see many business cases(like create comes after delete).I know kafka has consumer group and each partition is allocated to each consumer and ordering is achieved using this. ll definitely check session peek concept you mentioned in the video✌🏻

[@Matlus]

Thank you Sumit! I'm curious, how/why does your system have a delete request coming in before a create . You need to fix the root cause, rather than expect to fix this issue using a message broker. Unless I'm just missing something here :)

[@sumitmore4680]

@@Matlus oops sry. I gave loose eg to explain ordering concept. Let us take a better and funnier eg. suppose person A unfriends person B on FB. Person A writes FB status thinking person B won't see as he/she is not friends anymore on FB. In this situation, ordering of the events will be important (status update of person A event should not come before unfriend event of person A). I know this example is little funny and childish . LOL. but yeah similar situation can occur in business cases.

[@Matlus]

@@sumitmore4680 Hi Sumit, yes in that case (if the systems are distributed (separate systems) both events will need to routed to to the same topic and consumed by same subscriber. This will severely limit the scalability of the system however. Distributed systems (SOA/micro services) are "Eventually consistent" for this very reason. But I get your point, and there are some (related domain events) that will need this treatment.

[@sumitmore4680]

@@Matlus Yes. in topic we might have partitions based on user id and one consumer will only consume that. Things start to get interesting when distributed systems come in picture (and we have to make choice in CAP theorem). I saw few presentations where people thought of using kafka as db and then they related acid properties to kafka and all(sort of fun and interesting). Anyways, really excited for this series and looking forward 🙌🏻

[@sibasishsahoo2102]

Simply Awesome!!! Waiting for the upcoming videos on this. We have used message broker (msmq) for logging in most of the applications in our organization, but difference is there no local storage of logs. The win service reads queue and pushes logs to db. Do you suggest anything needs to be changed in the current implementation? Also there is one scinario where we have used IBM mq. The requirement is data needs to be formatted and synched between 2 different systems. Please let me know your thoughts on this.

[@Matlus]

Thank you Sibasish! The second scenareo you mentioned seems sound. For the first one, unless you've had trouble, I wouldn't change it. The pain points could be that the win service can't reach the destination and thus the local MSMQ fills up beyond its limit. If that has not happened yet then leave it be. Knowing where you've made a compromise is very important. NOT doing anything about it because its not needed is the mark of a good designer!

[@vishalgamji1324]

Good one, Shiv.

[@Matlus]

Hey Vishal, long time no see. Thank you!

[@vishalgamji1324]

@@Matlus Yeah, been a while. Looks like your follower base is growing. Good job. Very useful and practical content.

[@hansrudolf5849]

Just came across this video again :) Any reason why you first encrypt and only afterwards compress? Generally I would assume compressing first and encrypting after would yield you smaller request sizes. Still secretly hope that one day you will upload some new videos ;)

[@11r3start11]

Great topic, and as it's a quite popular a couple of years it`s good to have such detailed materials, thank you. Are you going to cover anything event-sourcing related?

[@Matlus]

Thank you Reslea! I wasn't planning on going that far. It's like putting the cart before the horse. I don't want folks stuffing their applications with cool new things they've learnt unless they have a really good reason to include a technology, pattern etc. Since this is an introduction, at least for now, I plan to have other videos that will help folks get started and playing with this technology.

[@aparnagarimella2564]

Hi Mr. Shiv Kumar, I am a big fan of your channel, the presentation and the content are excellent! I care about understanding the basics that drive any framework and love that your videos focus on that. I would like to ask your opinion on using Entity framework in production code (I apologize if you had already mentioned this in your previous videos, if yes, please post a link if possible). At work, our company is upgrading our current project with the latest and greatest frameworks and have started have started enforcing inline sql queries stored in class files as constant strings. We have only scratched the surface of the complex project which has thousands of stored procedures and having all that logic inside c# in the form of strings and is scaring me. The argument I received was they do not want to maintain a separate database project. They also did not give me a reason of not choosing something like entity framework where at least the query logic is readable in terms of c# statements and the relationships between entities is clearly defined. I hope you can help clarify this for me or at least point me in the right direction. Thank you!

[@Matlus]

Hi Aparna, thank you for your kind words. It is greatly appreciated. I would never use (And never have used) EF in any mid to large sized system. Most companies I’ve worked/consulted for have eventually had to remove EF and resort to using ADO.NET I would use a Database project in Visual Studio to maintain the database (this is recommended practice). This project also allows for producing diffs between say your previous schema that is in production and your current schema since then. I would strongly recommend using stored procedures rather than SQL in C#. Having stored procedures is so much easier to maintain. They can be debugged just like regular C# code. They are more efficient.

[@aparnagarimella2564]

@@Matlus Thank you so much for the reply. May I pester you to give maybe, a couple of reasons that totally broke the deal and caused the companies to remove EF? Has EF Core not solved any issues? I always like to understand why a choice was not made as I feel that is where true expertise/experience matters. Its easy to go with something new but it takes deep understanding to not go with something or when one choice is made over another. Maybe this could be a "Let's talk" video or a debate? I would also love to see a video on SQL server as well, recommended practices and gotchas both in development and unit testing. We have tons of business logic in stored procedures and I want to understand exactly what kind of logic should reside in a stored proc and what truly belongs in C#. I apologize that this comment is not relevant to the topic of this video so please let me know if there is any other way to post a question to you.

[@Matlus]

@@aparnagarimella2564 With regards to EF, primarily there were 2 big reasons (EF core doesn't resolve these and EF never will): 1. Performance 2. Developers can make too many mistakes when using EF (causing performance issues). There is a lot ot know and understand and the burden comes on code reviews and load testing (too far down the path). Eventually developers need to know and understand SQL anyway to better understand how to write the queries in EF. Given all of this, You might as well write a stored procedure and use ADO.NET directly (EF is just one more layer on top of ADO.NET). In some companies they use Dapper (A a thin layer on top of ADO.NET developed by StackOverflow) rather than drop down to ADO.NET (some people think ADO.NET is too low level/hard). I do plan to make videos on Databases and NoSQL But it will be a while before you see anything I'm afraid. As regards, Unit testing I have a playlist (4 or so videos) you might want to watch.

[@aparnagarimella2564]

@@Matlus Thank you sharing those reasons, I understand now. I have watched a few of the Unit testing videos, I will watch the rest. Thank you!

[@loyolastalin]

HI Shiv.. Will you please let me know, how to handle the dead letter/position queue in the real time scenario and what are the parameter to be considered?. How to handle the situation when the subscribers are down/ not able to process the message from the QUEUE [AGE / Depth]?

[@Matlus]

You don't handle scenarios when subscribers are down, because you don't know they exist. You may me thinking on pub/sub in an incorrect manner if this is your concern. There may be situations where you publish a message on one Exchange and subscribers publish responses on another Exchange. And you keep a tally of how many responses you received and from whom, but you're effectively tightly coupled to your subscribers. This is not a good use-case for Pub/Sub honestly. With regards to deadletter queues. There are many ways in which to handle them including manually checking your queues. It's all based on your scenario I'm afraid. The basic idea is that if a message was not able to be processed by a subscriber due to exceptions, they'll eventually end up in the deadletter queue allowing subscribers to get the next message (otherwise they'll never be able to move past that one message) while also having a way to see these messages in the deadletter queue and determine what t to do with them .

[@hansrudolf5849]

Seems like I have to watch those first ... :) I am curious what's your opinion on the cloud in general? (vendor lock in, data privacy, monthly cost, security, .. ) Listening to you it seems that your current employer makes use of the cloud quite extensively? ;)

[@Matlus]

Hi Hans, I think the cloud is the way to move forward. Not so much for individuals but certainly for enterprises. We don't typically consider the cost of us doing tasks that take us away from what we're supposed to be doing and if you're spending time baby sittings hardware infrastructure etc. The clouds alleviates a lot of that. The larger the company the more beneficial. The costs are obviously nothing (for large enterprises) as compared to what they spend today on buying, managing, maintaining their own hardware. Plus the up time guarantees, geo availability etc. are very difficult to pull off if you're doing it yourself. Privacy is well taken care of by all of the certifications cloud vendors have. You're also essentially in a secured VPN. That is, even though your in the cloud your resources can't be accessed from outside your network. As regards, vendor lock in.....we'll I can't speak to that but you're locked into so many things just just the cloud. Large enterprises don't have a lot of heart burn around this, I find. They're joined at the hip with som many vendors because hteir business depends on these relationships.

[@hansrudolf5849]

​@@Matlus Thank you Shiv for your answer - it's always nice to hear your opinion on topics ... :)

[@nirajdahal5019]

Hello Sir. Great videos. can you make a video on consuming 3rd party api in .net core . That would be great. :D

[@Matlus]

Hi Niraj, did you have a particular API in mind? What specifically would you like me to cover?

[@angelmejia8452]

messi brokers