System Design: Why is Kafka so Popular?

Get a Free System Design PDF with 158 pages by subscribing to our weekly newsletter: bit.ly/bytebytegoytTopic
Checkout our bestselling System Design Interview books:
Volume 1: amzn.to/3Ou7gkd
Volume 2: amzn.to/3HqGozy
The digital version of System Design Interview books: bit.ly/3mlDSk9
Animation tools: Adobe Illustrator and After Effects.
ABOUT US:
Covering topics and trends in large-scale system design, from the authors of the best-selling System Design Interview series.

[@ByteBite101]

You inspired me to make a good explainer. Thanks. As always, great explanation❤

[@jonathanvargasv2009]

Thanks for the summary

[@DinoRossYT]

I mainly use kafka for producers apps to consumer/processors microservices. That was a whole level up to let connect components in my dev exp. But i will take those examples into account, they seems promising if we have to let go logging services like plausible or sentry.

[@gus473]

Great overview! 😎✌️

[@raj_kundalia]

Thank you!

[@tobbymarchal3140]

Please make video about redpanda vs kafka

[@ILyaCyclone]

Can we get Flink overview, please?

[@rajalalwani502]

How does RabbitMQ stream performs in comparison to Kafka? Can it be considered as alternative to kafka in any of the usecases?

[@Nhuzaa11]

Which software do you use for this cool minimalistic animations

[@ByteBite101]

I use After Effects to make such explainer videos. I think, he uses that as well.

[@rajatgoyal715]

He has mentioned the Animation Tools used in the description of the video: Animation tools: Adobe Illustrator and After Effects.

[@joelamltc]

How about compare to Solace?

[@azizmalik5224]

Pre-migration reconciliation is unclear to me. Can anyone explain more?

[@saitriesyoutube]

I believe Kafka is acting as an abstraction layer. Either v1 or v2 can produce events or data because the downstream consumers consume it via Kafka, and to those services, it doesn't look like anything has changed.

[@LakshithaKarunanayake]

Your lecturing speed is so high that I might need a distributed event streaming platform to understand it...

[@hemanthkumar-xf8ki]

Hahaha, you understood Kafka.

[@zojirushi1]

Is lecture speed too fast or slow?

[@pavelpikat8950]

No mention of Event Streaming or Event Bus?

[@gigakoresh]

Kafka is way overkill for that. Learned that the hard way. Only makes sense if you already have to maintain a kafka cluster or buy it as a service for something actually data-intensive. Even then, by itself Kafka is a huge pain the ass to configure and maintain. You need a management system on top of it like Confluent or Aiven, if you dont want to fiddle with its terrible CLI tools and build ACLs all day. And those cost an arm and a leg. Also Kafka rarely breaks, but when it does, it's a catastrophe. There is no good way to quickly and automatically recover a broken cluster, and it's very easy to fuck up. I can give you one example - we used a 3-DC cluster where all communication between clients as well as brokers to each other was handled by IP lists. DNS was never used. Then at some point the DNS proxy we used got busted and cluster ended up in split brain state. It was completely incomprehensible why. We had to do network packet monitoring, when we found out that even though the broker didn't use a DNS, it still had a call stack failing silently when contacting brokers on outgoing calls. So all messages coming to the broker were going fine, but all messages going out were failing after a failed DNS lookup for... something that the broker wasn't even gonna use! So other brokers thought the broker was alive and well, and the broken broker thought it was broke (i mean alone, sorry, couldn't resist), constantly trying to assign itself master since it thought the rest of the cluster was dead. Took a whole day to fix and you can imagine the damage since the whole cluster was affected... And this is far from the only example. My experience with Kafka is actually that it's distributed nature is a tool for scalability, not reliability. As far as reliability goes, it does the opposite, each new broker adds more ways for the whole cluster to break. So yeah, Kafka is powerful, about as much as a bulldozer is powerful. If you only want go to shop and back, maybe just take a regular car instead.

[@msatanzeel]

He mentioned about event streaming (like telecasting live events)

[@adamjones9600]

@@msatanzeeldifferent. Episode 433 of SE Radio might be of interest to you though.

[@balfiman]

One morning, when Gregor Samsa woke from troubled dreams, he found himself transformed in his bed into a horrible vermin.

[@Dr_Larken]

Free Kafka cookies!

[@akash-kumar737]

😊 it was yummy 😋

[@BohonChina]

Kafka design is out of date for cloud native environment, there are many other alternatives for kafka right now such as confluent, red panda, AutoMQ and Pulsar.Compared with these MQs, its performance,latency and storage management is not good enough

[@millouwmills367]

Confluent = Kafka isn't it?

[@JTWebMan]

For the love of god please pick something else unless you really need kafka's scale which 99.999% of companies and problems do not.

[@msatanzeel]

But why? What's the problem with Kafka, and when you mean other services, do you mean third party fully managed services like AWS Kinesis streams?

[@abdullahcosgun]

Kafka is a behemoth of a system. It requires expertise to run and manage. You need minimum of 3 nodes to have a healthy cluster

[@dinoscheidt]

Red Panda is Kafka complaint and has nothing to do with java and the dreaded zookeeper causing most of the problems of Kafka. Apache Kafka is legacy tech, from an era when CPUs and hard drives where slow, and therefore worth to be replaced with event steams, by-directional message brokers, in memory tooling like redis and distributed relational databases for new systems started today.

[@MrAnshulji]

So, RabbitMQ then? or Amazon SQS?

[@dinoscheidt]

@@MrAnshulji It is only important to not be a purist. Mix and match. For example: If two people want to buy the last airline ticket at the same time, you have to implement complicated architecture (outbox inbox, dead letter queue etc ) patterns to handle that with messages. For the buy button we simply use gRPC, which is bi-directional and blocks if you are the person that came a nano second too late. Think in architecture types not in products like RabbitMQ vs Redis vs Amazon vs Azure vs Kafka vs GCP …...

[@rsy1708]

I am not a native English speaker, but my English has already reached C1 level, and I can understand US Americans and British people very well, even several dialects. However, it's still challenging to understand people from other countries speaking with an accent. What I don't understand is why so many people don't care about learning pronunciation. Learning a large vocabulary may take years, and mastering the grammar might take months, but learning good pronunciation should only take a few weeks. Having good pronunciation would not only result in more people being able to understand you but should also lead to fewer misunderstandings. Even for native speakers, it's easier to understand someone with good pronunciation. The less effort you make in speaking correctly, the more effort your listeners have to make to understand you.