16. System Design - Distributed Messaging Queue | Design Messaging Queue like Kafka, RabbitMQ

➡️ Notes: Shared in the Member Community Post (If you are Member of this channel, then pls check the Member community post, i have shared the Notes link there)
➡️ Join this channel to get access to member only perks:
/ @conceptandcoding
This question in interview can come with different ways like :
Design Distributed Message Queue,
Design Kafka System,
Design RabbitMQ system,
How Messaging Queue works.
Chapters:
00:00 - Introduction
01:32 - Messaging Queue and its Advantages
07:49 - Point2Point and Pub/Sub Pattern
10:51 - Kafka Messaging Queue in Depth
39:07 - RabbitMQ in depth
To. connect 1:1 with me: topmate.io/concept_coding
To contribute for the channel UPI id: shrayansh8@oksbi
Join this channel to get access to perks:
/ @conceptandcoding
#systemdesign #softwareengineer #interview #amazon #hld

[@santoshkumar-qf3ke]

teaching is an art, he literally said … “tell me” ….. reminded me of my math teacher

[@ConceptandCoding]

loved your comment buddy thans a lot means lot to me

[@shaijil09]

I have worked in multiple systems with Kafka as a messaging broker. This video is an excellent tutorial ❤. I would love to see more tutorials that explain concepts like timeouts, effects on networking latency in polling, etc.

[@jaideeppyne1880]

This is one of the best Kafka videos I have seen on KZbin. You have taught so nicely here that I am looking for a playlist only of Kafka in your channel. A few things that were best -> format of the video, you started with a small box and kept on adding pieces, separately explained broker, topics, partition, consumer, consumer groups, then cluster, and zookeeper ... this gave an idea which is what and small functional components kept adding up and formed the full system. BEST! Few things I think can be added to make this a true masterpiece and no need to watch any other videos -> some twisted cases maybe.. complex cases with multiple partitions, multiple machines, and different partitions of the same topic in different brokers. Several other cases and a bit more extension of this in part 2 would be great. These questions are asked sometimes by the interviewer to test the knowledge.

[@ConceptandCoding]

Noted, thanks will add more videos on Kafka to covered more usecases

[@manansahlot9906]

Thank you so much for sharing high level architecture of Kafka. Too much informative enjoyed the video!!

[@ConceptandCoding]

Glad you enjoyed it!

[@saumyachaurasia7350]

Wow...the explanation is so on point. Thank you so much for this valuable content.

[@ConceptandCoding]

Thank you, Saumya pls do share it with your connections ☺️

[@DeletedUser502]

Concise form, covered all the kakfa topic.

[@sudhirdharmadhikari1938]

Too good ! Saved several hours of reading from the book

[@ConceptandCoding]

Glad it helped!

[@nobodyknows228]

I have watched more than ten videos on this no body explained it like you. Thanks a lot for making such content.

[@ConceptandCoding]

thanks a lot

[@468hitman]

Very well explained

[@novascotia2015]

love you sir. thanks for giving these classes as free

[@ConceptandCoding]

Welcome buddy, pls do share it with your connections ❤️

[@tonystarc9567]

May I confess to you that "You are my hero". Thank you for everything that you are doing. Knowledge empowers the society.

[@ConceptandCoding]

Thanks a lot buddy, means lot to me

[@ziyinyou938]

Good job mate!

[@mdnayab5839]

Thanks a lot, that was so much helpful. It cleared all my doubts😀 Please keep up the good job!!!

[@ConceptandCoding]

Thank you

[@makarandthakare9174]

Thank you for the black background. Really helps the eyes.

[@ConceptandCoding]

Yes got lot of feedback for this, so switched to Black background

[@anketpatel2312]

Perfect one , you can come with implementing it in a project.

[@ConceptandCoding]

Thanks

[@GulabSingh-iw5nr]

Amazing video , great explanation 44:29 he mean *rabbitmq* works on push approach

[@shamstabrez2986]

Dkhne se phle hi comment karta hoon I truly blve is bande kamal ka content diya hoga is session mein

[@ConceptandCoding]

Thank you very much Shams❤️, pls do comment after watching too if you liked the video and found it informational , really wanted to know

[@subhamacharya4472]

Great video on messaging queues 👍😃

[@ConceptandCoding]

Thank you, pls do share it with your connections Subham ☺️

[@tejasshaha6629]

Just amazing video. Thanks a lot ❤

[@ConceptandCoding]

Thank you

[@heykalyan]

awesome didn't know about this requeue concept Sunday sorted😃😃

[@ConceptandCoding]

Thanks buddy do share it with your connections too ☺️

[@makarandthakare9174]

Currently on 11th video, all amazingly explained. Thank you.

[@ConceptandCoding]

Thanks buddy, hope by the time you will come to 16, i will add couple of more videos till then :)

[@makarandthakare9174]

@@ConceptandCoding LLD and HLD complete ✌️😀

[@akshitamadaan7867]

Very valuable content. Thanks a lot🙏

[@ConceptandCoding]

Thanks a lot

[@JoJoy]

Excellent! Thanks!

[@JoJoy]

One improvement you can make for your future videos is to understand the use of "read" (present tense) and "read" (past tense). You are superb in spite.

[@gauravraj2604]

Insightfully explained. Thank you for your contributions. However, I feel you should have discussed acknowledgement part which could have provided more in-depth understanding.

[@prasoonkushwah9558]

very detailed one , keep going thanks

[@ConceptandCoding]

Thanks

[@yaseenshaik2284]

Very useful content. Thanks

[@ConceptandCoding]

Thank you

[@ujjalroy1442]

Best ever i have seen...thnxxx

[@ConceptandCoding]

Thanks Ujjal, pls do share it with your connections too ✌️

[@ujjalroy1442]

👍👍 sure

[@prasannapm3220]

Nice!

[@barwalgayatri4655]

Best Explained

[@ConceptandCoding]

Thank you

[@suheabkhan2546]

Zabardast

[@ConceptandCoding]

Thanks

[@rupeshjha4717]

Thanks a lot for this awesome video! Happy to see your channel growing now! It would be also helpful if you can guide how to decide how many partitions , concurrency, retentions etc should have in kafka ? There is no such docs or tutorial on this estimation of these things, i have struggled a lot to figure out this. Hope you will pick up.

[@ConceptandCoding]

Hi Rupesh, first of all thank you. How many Partition should have, it depends upon traffic volume ( generally partition number is in odd like 3, 5,7 etc) but how many you needed depen upon traffic and it can be increased later also, so to answer your question there is no fixed number you can start with 1 and grow as per need

[@prafullaingole6122]

Nice explanation. A real world practical example would have been great to connect the dots.

[@ConceptandCoding]

Yes i will cover 1 topic on that too

[@AnuragMishra-ws4zc]

Sir in system design interview we should only know overview of it (Kafka,RabbitMQ)or we should also know how to integrate it to our projects means internal working also.

[@ConceptandCoding]

No internal working is not required, as in 45 mins interview thats not possible even Interviewer might not know how to integrate without reading the documentation

[@yc98]

Have doubts about the part where the message is stored in the dead message queue after retries in kafka, you said after certain retries it goes to the dead msg queue, but after a few minutes you explain that message is pulled by consumer itself in Kafka, so if the consumer is down it will not pull, and if it doesn't pull - message retries doesn't make sense. Can you please explain.

[@rajeshreegavhale3227]

Best kafka video

[@ConceptandCoding]

thanks

[@barwalgayatri4655]

Please make Series on How Multiple Microservices talk to each other using Kafka or Rabbit MQ ?

[@anoopamz]

perfect

[@mrinalpandey54]

Thanks a lot for such a great article. I have few doubts 1. Can kafka implement both PUB/SUB or Point 2 point? Which is by default available with kafka? 2. In last u explained about RabbitMQ and same message getting broadcasted to multiple queues. how consumer will know wheteher the message has alredeay been processed by a different consumer? what is the business use case when we want the same message to be processed by different consumer? 3. Does kafka use Point 2 point?

[@ConceptandCoding]

Kakfa is by default distributed Pub/Sub architecture. And it does not care whether same msg is processed by different consumer (in some other consumer group) or not. USECASE where same msg multiple consumer need is, Take the same example of Cab services (in which they are sending their location every 10sec). One consumer is Dashboard for all Car location in a particular area. Another Consumer could be logs Application, which keep logging everything. Another Consumer could be the Real time Cab movement.

[@atulsingh6624]

i would appreciate if you come up with video implementing the kafka using psuedo code.

[@rhythmjayee9707]

🔥

[@youKnowMe4real]

whats the point of different brokers if we have replicas of partition as ultimately both in sync hence both brokers will get full queue at same time??

[@ankitgupta-ph4nk]

Very easy and well explained, Any practical reference(Code wise) if you can share for this messaging will really helpful

[@ConceptandCoding]

Thanks and noted

[@vaibhavjain1124]

First great content Shreyansh. Kudos for that. I would like to add a bit clarification from official documentation on offset management. This offset mechanism where information about offset is stored in zookeeper znode is and old architecture. As per the latest version kafka has already moved out of dependency over zookeeper znodes to maintain the offsets, Kafka introduced an internal topic called "__consumer_offsets." The offsets are now stored as messages in this topic. Brokers replicate this topic for fault tolerance.

[@rtrivedi39]

Thanks, It gives me good understanding of both of the queues. Can we add here AWS SQS too.

[@ConceptandCoding]

Thanks and noted

[@AkashKumar-yi2jy]

As you stated, if a partition is full, data is automatically moved to another partition. How will the hash-function know that a specific partition is full? How does Kafka handle partitions during rebalancing? How are network latency, data consistency, and data synchronisation handled when Kafka is used to replicate data across multiple data centers or geographic regions?

[@ConceptandCoding]

Hi Akash, in Kafka msgs are also deleted from queue periodically. Generally it has 2 criteria: - Set Retention time in topic. (After that time, msg will automatically deleted) - Size limit, once the size limit is reached, it will purge all msg whether it's read or not. So same partition can be reuse again. Also Publisher can implement Acknowledgement approach too for make sure that Msg is actually added in the question. If Ack=0 means Fire and forget. Publisher sent the msg but it might not added in the queue. Ack=1, means msg is accepted in one more partition.

[@ArunGupta-ig2hc]

When you say push based approach, and messages are pushed to consumers, Does that mean, bidirectional connection is established between the queue and consumer in RabbitMQ ? If not, what protocol is used underneath for push mechanism ?

[@vinny1577]

I agree, with all this information covered, it would have been nice to talk about how push and pull are implemented.

[@abhishekpattnaik8531]

Hi Shrayansh, Suppose there is one rabbitMQ queue and there are multiple replicas of a service . The service is a consumer of this queue. In this case, will both the replicas consume the same message simultaneously? If yes then how to avoid such scenario?

[@romechax]

GREAT VIDEO !! May be just some grammatical errors while explaining. If that's improved, it can be world class .. keep it up

[@ConceptandCoding]

sure i will improve, thanks for the feedback

[@vikramreddy7586]

Did I miss the part where queue is designed, the title Design Messaging Queue like Kafka, RabbitMQ seems misleading to me.

[@bhaskarbhakat5689]

Do we really have dead letter queue thing in kafka? I guess we can only have this sort of functionality by adding a new topic called dead letter topic. Please correct me if I am wrong.

[@satyamtiwari5515]

Hey, Could you please share your hand written material i-e oneNote/ipad link too for the reference.

[@HariSarvotama]

amazing video sir ! have been following you since many months now . any specific reason why did you switch to english instead of hindi ? i liked hindi videos more in general

[@ConceptandCoding]

Hi Aditya, got 100s of msg that they also want to learn but do not understand Hindi, so after thinking a lot i decided to switch to English. - My English is very basic Kam chalu. So anybody can understand. - 2nd point is more than language, way of teaching is imp, and that i kept the same. So i moved on

[@sitronco]

Thank you so much this is awesome. One quick question: Do you have the notes available anywhere?

[@ConceptandCoding]

Notes are available in description section, attaching below too: notebook.zohopublic.in/public/notes/u3i1s522a981ed32d48bcbb0b940ee3d58f22

[@karanbangia14]

Hey, i have a question lets suppose, there are 2 partitions for topic and there are 2 pods running on kubernetes which are pulling messages, so 1 pod will read from 1 partition, now lets suppose there is a surge in message so new partitions will be created, how will my pods know about this? adding partition based scaling is the only solution as that partition will remain idle?

[@ConceptandCoding]

I don't think, Kafka support automatic scaling with partition of a topic, we have to provide the Count of partition while creating the topic. When we have to increase or decrease the partition, as what I am aware of, it create new topic with new count and stream the data from original to new topics which is created and update the zookeeper. But let me check in morning again. But this is what i know

[@aartikelkar397]

Thankyou for the valuable videos! I have a doubt at 27:00 - you mentioned that consumer2 will start consuming msgs from partition1 but earlier you said that each consumer of a CG will consume msgs from different partitions. Here what if the C2 is consuming msgs from P2 already?

[@brahm_and_coding]

when c1 goes down the consumer group will pick the next available consumer to resume the processing

[@aartikelkar397]

@@brahm_and_coding thanks

[@sumitbasu5146]

Hi Srayansh, I have one Qs. Suppose, Application 1 and Application 2 acts as a consumer 1 and consumer 2 and it is listening to Topic A message. Both are belong to one Consumer group. Now, when message published and suppose Application 1 consumed the message then will the Application 2 also consume the message?

[@ConceptandCoding]

If 2 different applications but with same group Id means belongs to same group. And both are reading the same topic, then they can not read same msg. Because as i mentioned, inside topic there are partitions which consumer read. So consumers in same group, can not read same partition. Therefore we can say, same msg can not be read by both consumer those belong to 1 consumer group

[@suranjanachattopadhyay4979]

Could you please elaborate the retry mechanisms that should be used while using kafka?

[@ConceptandCoding]

In video i did, it maintain a sequence no right

[@balakrushnabehera2800]

Very nice explanation. Could you please tell me about consumer group if consumer group 1 has 4 consumers does it mean that those 4 consumers does the same work like they are replicas of a particular consumer or they can be diff. Consumers doing different jobs. Basically I am asking while creating consumer groups on what basis we divide consumers to different consumer groups like they are grouped as replicas to each other or consumers doing different works can be grouped to one consumer group??

[@ConceptandCoding]

Each Consumers are not replicas. Depending upon company needs different consumer application might be created, let's say one app need cars location data to create dashboard for users. But other need the data for doing something else. If both needed same data they need to be put into different consumer group

[@balakrushnabehera2800]

@@ConceptandCoding So you are saying that if two consumer applications need same data then they need to be placed in diff consumer groups as keeping them in same consumer group the applications cannot read the message concurrently??

[@dakshtomar9996]

One question:- if a consumer goes down in Rabbit MQ then how is a new consumer assigned to read from that queue?

[@justlc7]

Hey Shrayansh, could you do a similar video for GCP Pubsub? Thanks.

[@ConceptandCoding]

noted

[@Anjali_Ranveer_Singh]

what happens when the dead letter queue overflows?

[@Telugu_lifestyle]

Hi. Thanks for the Wonderful explanation. Please help me in getting clarity for below one's 1. I hope the P2P queue can be achieved by pub sub (kafka and Rabbitmq with Direct exchange technique). If not kindly help me in knowing it 2. How key (hash) in sender payload decide to push the message in kafka partition topic. Does the key's hash is generated based on partition information? Thanks in advance and thanks for the effort ☺️😊

[@AkashKumar-yi2jy]

We can design a topic or exchange that is unique to every producer-consumer combination to establish a P2P queue where each producer has a queue to send messages to and each consumer has a queue to receive messages from. This is possible in RabbitMQ by using a direct exchange, which routes messages to queues based on a routing key. Each producer can have its own queue, and each consumer can use its routing key to consume from a specific queue. This is possible in Kafka by using topics, in which each producer publishes to a specific topic and each consumer subscribes to a specific topic.

[@AkashKumar-yi2jy]

The hash value of the key is used to determine which partition the message is assigned to. The MurmurHash2 algorithm is used to generate the hash value, which is not directly based on partition information. The number of partitions in the topic, however, has an effect on the calculation of the partition to which the message is assigned because the hash value is taken modulo the number of partitions.

[@ConceptandCoding]

Thanks Akash for the input. And regarding the first point can P2P be implemented via PubSub like Kafka. By putting one consumer and Consumer group, we can get the P2P behaviour. But in general Kafka is Pub Sub pattern

[@animeshbiswas2905]

hello sir LLD can be done with javascript (also learned oops ) because this my first language or i need to learn java?

[@ConceptandCoding]

Hi Animesh, LLD is OOPs concepts only. I am not familiar with javascript, see if all the OOPs fundamentals is possible with Java script then you are good to go)

[@mehulanshumali]

Possible to share the slide ?

[@khushboojain3618]

Hi Great videos 👍🏻What is the software used for system design interviews

[@ConceptandCoding]

One note and wacom

[@ADITYAKUMARking]

Hey, Let’s assume in consumer group, there are two consumer 1 and 2. If consumer 1 went down, then will consumer 2 start reading from the partition which was assigned to consumer 1? What if consumer 2 was already assigned assigned a partition, what will be behaviour now? Also judging from the design here do consumer group consists of single service(with multiple instance/pods)?

[@ConceptandCoding]

Consumer 2 will now take care of both the partition of a topic.

[@mrinalpandey54]

@@ConceptandCoding hi but in the video u taught that at a time only 1 consumer can interact with 1 partition. So is this an exception case or do we have any configuration through which a consumer can talk 2 partitions? Can a consumer read from more than 1 partition?

[@ConceptandCoding]

As per Kafka documentation. If there is 2 partition and 1 consumer, then 1 consumer can read from both Partition. But if there are 2 or more consumers active and 2 partition, then correct configuration should be that each consumer should mapped to different partition

[@sharmath101]

Suppose there are multiple consumers from same group and they are reading message from same topic A.what if they want to read same message msgA which is present in Partition0. Does producer push duplicate message to each partition in the topic. How this can happen.

[@ConceptandCoding]

No that's not possible as per Kafka documentation. 2 consumers in one consumer group can not read from Same partition. But same msg can be present in different partition of a topic. That logic need to be present at publisher, to publish same msg one for say Partition0 and another partition1.

[@sharmath101]

@@ConceptandCoding Here i am considering a consumer to be suppose ServiceA , ServiceB . Both need to work on msgA. so msgA will be pushed to both the partition in a topic. Am i thinking right?

[@ConceptandCoding]

@@sharmath101 right. But generally if ConsumerA and ConsumerB both need to work on same partition, the correct way is they should be part of different consumer group that make more sense. You are right

[@NileshKumar-fz9ok]

is AWS SQS another example of Distributed Messaging Queue?

[@ConceptandCoding]

Right

[@sauravgupta7415]

Kafka itself does not have a built-in dead-letter queue

[@Abdulrashid22067]

I have one doubt .. kafka is pull based then why will zookeeper take care of assigning topic to a consumer 2 when consumer 1 goes down? Isn't it consumer's responsbility

[@ConceptandCoding]

This is configuration based logic, as per my understanding, when consumer group is created, it configure the strategy by which it decide which partition to assign to which consumer, which strategy to choose when any consumer goes down. And strategy i think is Kafka platform code(called partition assignor). Let me double check where does this code logic or strategy resides.

[@kalyanbabuakula2662]

Hi sir, what happen if zookeeper is down will there be a replica for this

[@ConceptandCoding]

Yes Zookeeper is also a Distributed system, one goes down another comes up. But did not cover it else video length becomes very big

[@HellGuyRj]

Wait a sec, RabbitMQ works on push approach right? Why did you mentioned kafka works at push approach near 44:30?

[@ConceptandCoding]

Oops sorry my bad. I wanted to say RabbitMq. RabbitMq - works on Push approach Kafka - works on Pull approach. Thanks for pointing out Yash.

[@HellGuyRj]

@@ConceptandCoding no worries, mistakes happen. Btw you are doing absolutely godly. Thanks for the content. We

[@tech_knocking8440]

Awesome content, but video length could be long, 45.12 is a very short period of time to understand complex topics like Kafka and Rabbit. Sometimes I felt you were in a hurry to wrap it up, please take it in a positive way :)

[@ConceptandCoding]

Thanks for the feedback Ratnakar. Taken this feedback

[@ankitgupta-ph4nk]

I am watching it again.. Just one question.. Let say we only have one consumer group and 2 consumer in it.. And we have 2 partitions inside 1 broker. You also said in a same consumer group each consumer assigned to a different partition. Now let say if 1 consumer goes down while reading from partition 1 at offeset 5.. And assume 2nd partition is reading by consumer 2 continuing without failure.. What will happen in this case for partition 1

[@sam-um5wo]

In that case where consumers are less than partition kafka will rebalance the partition and consumer 2 will read from partition 1 for sometime and keep on doing the rebalancing again and again

[@ankitgupta-ph4nk]

@@sam-um5wo Ok, then what about consumer 2 which already reading from partition 2..

[@sam-um5wo]

@@ankitgupta-ph4nkIt will do that In round robin fashion sometime from 1 and then sometime from 2 .Read the Kafka documentation.Thats what is consumer rebalancing in kafka

[@ankitgupta-ph4nk]

@@sam-um5wo I got this thanks man.. Just one more thing asking out of context... I want to learn Low level design .. Let say for pizza store or any parking lot.. How to structure all classes and what best design patterns we can use Do we have any resources online to thoroughly got those concepts..

[@BRIJKISHOREDUTTBCE]

Why does two consumers within a consumer group can not read a same partition?

[@ConceptandCoding]

This is necessary to maintain message order and prevent duplication buddy. This design ensures sequential processing within a partition, avoiding complexities associated with concurrent consumption from the same partition by multiple consumers.

[@suheabkhan2546]

Hi @shrayansh Even after repeated requests, you are not sharing the one note links for notes😢

[@ConceptandCoding]

Hi Suheab. When i convert these One note pages to PDF, it's not in format at all. No one can read and understand properly from it. Sharing those rough kind and not formatted notes is not a good impression. One way for me is, to write it properly and then share, but again it gonna consume some time. Let me reach out to some team members and see if anyone while watching video make notes. I will try to get that.

[@suheabkhan2546]

@@ConceptandCoding you can directly share this one note link by making it read only

[@subhampandey7374]

What if the queue goes down for rabbit MQ?

[@ConceptandCoding]

RabbitMq also has the same concept of broker, leader and follower. So it should behave in the same manner as Kafka

[@tejasvaidya8464]

Sir can you please provide this document for reference

[@ConceptandCoding]

Noted, will try to upload on gitlab

[@shubhamsinghrana9262]

could you please share notes on same

[@ConceptandCoding]

Check description once pls,, if it's not there, i will add buddy

[@prashantgupta6160]

Why can't we use db, why message queue

[@ConceptandCoding]

actually both are used, queues are used to store the message temporarily and when all retry finished, it does store the failed msgs in DB. So multiple distributed workers can easily work on the queue and its easy to build event driven architecture using queue. DB is mostly for storing permanent data and doing some complex operation or queries.

[@sorrybabu9744]

One thing is wrong here partiotion is not the leader, Its Broker which is the leader in the cluster. Dont delete the comments.

[@ConceptandCoding]

Leader and follower are the concept of Partition. One is leader and one or many are followers.

[@ravikant-hi8mz]

To clarify, there is a leader broker machine for a particular topic and partition combo.

[@pawanverma2575]

Broker is not leader it is elected as Controller which is done by Zookeeper and the leader is one of the partition which is elected by Controller

[@sbera87]

Bro, be concise please. Too much repetition of words