When to Use Kafka or RabbitMQ | System Design

"If a consumer disconnects before committing the offset, Kafka will automatically send the data to another consumer." Actually, this part is a bit misleading, as Apache Kafka doesn't 'send' anything because it's a pull-based model which means when a new consumer is assigned to a partition, it will start reading data from the last saved offset for that partition. If the previous consumer didn't save its progress, the new consumer will start from the last saved point and might process some messages again.

[@interviewpen]

I can see how this was misleading. In terms of data flow, the message moves to an available consumer, but yes the consumer first must poll for new records. Traditional queues work this way as well. Thanks for clarifying :)

[@pratikshitsingh415]

Awesome explanation on the difference of use cases. I've worked with kakfa but kind of new to RabbitMQ and this video helps me getting the brief insights on these. Thanks

[@interviewpen]

Glad it helped. Thanks for watching!

[@Fikusiklol]

Both systems are designed to decouple. Both systems have routing/filtering/processing capabilities, that either built in or part of their eco system. Main difference is that kafka is designed for events, rabbit is for commands. And ordering/persistence associated with those.

[@muhammedashkar692]

Nice content. I was doing a microservice project where I had chosen Kafka over RabbitMQ without actual knowledge of the use of kafka. I needed the traditional RabbitMQ behaviour for my current microservice project, and I thought Kafka would be better for that as I heard it had high throughput. now I know the why's. gonna go with rabbitMQ.

[@interviewpen]

Nice, glad we could help :)

[@mageexian]

This is well articulated, 🙏

[@interviewpen]

Thanks!

[@atulshah1045]

Great breakdown of the queues and side-by-side comparison. And I can’t believe I’m the first one to comment here!

[@interviewpen]

Thanks for watching!

[@eduardmart1237]

Nice video

[@ryzlesalt]

this is a great summary of them!

[@interviewpen]

Thank you!

[@debangshubanerjee1311]

Excellent video, one request can u compare Solace Queue also with Rbbit and kafka?

[@whoknows3679]

Kafka can be used as queue tho, sharing the same consumir group and increasing the partitions as many consumers needed to balance messages

[@interviewpen]

It certainly can, but that doesn't mean you should...it's really not designed for that. There's a lot of strange issues that arise from doing this--you need lots and lots of partitions, group rebalances can cause message duplication, brokers will time out clients whose jobs run long, etc.

[@AlanD20]

​@@interviewpen What you are saying is misleading, Kafka indeed is built for both pub-sub and queue systems. Kafka is meant to handle loads of partitioning and in fact that's the whole purpose of kafka. It depends on your configuration to increase group rebalancing and broker timeouts. There arent any harm in doing these. Kafka configurations are very flexible to adapt to these models and they can cover the majority of cases, it's mostly up to how you configure it for your use cases. Not sure what you mean by "group rebalancing can cause message duplication", if a message hasn't been committed, it can be picked up by another consumer after the group rebalancing, and if your implementation does not handle idempotency when processing messages, the issue here is the implementation not kafka's fault. Even if you have database records, you can still handle these cases with transactions. And the whole idea behind the consumer group is to act as a pub-sub and as a queue depending on how you want to use it. Hope this helps!

[@VaibhavShewale]

ooh damn, it was interesting

[@interviewpen]

Thanks for watching!

[@0xtz_]

wait I can't imagine that's free content 😮 well explained 👏👏

[@interviewpen]

Thanks!

[@OutreGeek]

Brilliant! Glad I chanced upon this gem. My current project has a requirement to process real-time location co-ords continuously and by different consumers. I now fully understand why the Architect chose Kafka over alternatives. 👏

[@interviewpen]

Sweet. Thanks!

[@shis10]

Amazing video 💯

[@interviewpen]

Thanks!

[@matveyshishov]

I really like your videos (and even bought your course because of that), but this one seems to be lower level than your usual standards. You make it seem like the difference is mostly topological, or maybe API-wise. But in reality the systems are very different in purpose and use cases. RabbitMQ is an implementation of AMQP, full stop. Just go read John O'Hara's article "Toward a commodity enterprise middleware". Being an AMQP, all it does, and nothing else, is delivery of messages to the specified recepient. Again, full stop. As soon as the message is in the memory buffer - it's done its job. Kafka, on the other hand, is a CDC on steroids. A distributed WAL. There are no messages, therefore, no addressee, there are only "data changes", which we usually call "events". I could wax poetic for much longer, and do comparisons with this and that, but honestly, I don't see the point, as we live in the time when you can literally ask the person who created the system what they wanted to do ;) .

[@dan_le_brown]

Thanks for the clarification

[@interviewpen]

Yeah this is a good explanation of the implementation differences between the two systems. This video just covered the practical use cases and functionality, so thanks a lot for sharing this perspective!

[@augustinemunene3469]

would you kindly do how to make a backend as a service system design

[@interviewpen]

Yeah we might do that soon :)

[@shauryavardhansingh7917]

Give me a video on kafka and nats

[@bhanuprakashrao1460]

Are consumers in Kafka do polling to get the events or is it a push based model, where Kafka pushed the events to consumers? For traditional queues, I know that it is a pull based model, where identical consumers (i.e replicas) do polling.

[@interviewpen]

Kafka consumers poll for new records (same for traditional queues).

[@ashutoshpanda4336]

There is so many things wrong this video. Kafka doesn't directly fan out. Each consumer is assigned to one or more partition based on the configuration of kafak. Once a message is read and it doesn't send to all the consumer since all consumer would be assigned to different partition. This is just misleading. I don't know why people aren't calling it out. and There is another comment also pointing out "Kafak doesn't send anything it's pullbased" was there which is wronngly mentioned in video.

[@interviewpen]

That’s not true. Each message delivered to a partition is fanned out to every consumer assigned to that partition. And yes, consumers poll for new messages, as with traditional queues. This might help: docs.confluent.io/platform/current/clients/consumer.html

[@jephthahsnr6558]

🔥🔥🔥👌🏾

[@interviewpen]

Thank you :)