Apache Kafka: a Distributed Messaging System for Log Processing

[@gkcs]

Thank you for watching. Also, NEW EPISODE: interviewready.io/learn/system-design-course/building-an-ecommerce-app-1-to-1m/7-the-case-of-the-expensive-cache

[@ashutoshsamantaray6596]

zookeeper is no longer used in the latest version of kafka. The newer version is Kraft using consensus protocol among brokers. Makes it more scalable

[@gkcs]

Yes, they use RAFT now 😁

[@FakeDumbDummy]

way better than zookeeper for managing live clusters...

[@kidkool27]

Wow I was just out of college when I first found your channel. I’m a fresh senior now preparing for interviews, great video!

[@gkcs]

Glad to read this, cheers!

[@sarwajeetbharti2856]

Great Explanation with lovely animation !Thanks Gaurav Bhai !

[@sunnykumarsingh7039]

Such a simple yet elegant video Gaurav!

[@tayyabachowdary1007]

Have been following you since long and i am so impressed with the consistency and humility. Yoir explanations are simple and you seem like a college mate we are getting to learn from.thanks for all the free content

[@gkcs]

Cheers!

[@WinnersDontWin]

Been trying to study this for the best to weeks, and this is by far the best youtube on the topic

[@gkcs]

Thank you!

[@rajkumar-in3sk]

6:58 If I remember correctly, ZooKeeper uses the Zab (ZooKeeper Atomic Broadcast) protocol instead of Paxos for consensus. Starting with version 2.8, Kafka no longer relies on ZooKeeper for leader election. Instead, it uses KRaft (Kafka Raft), Kafka’s implementation of the Raft consensus algorithm, for managing metadata and coordination internally.

[@TheCodingChamelion]

Tomorrow is exam and this video was saviour for the kafka topic

[@SaiBharatBatchu]

It’s a great video. It covered the features of kafka well. I feel the way you explain a problem and explain how to overcome it might not be obvious to the listeners that kafka automatically takes care of these. For eg, you said multiple consumers might consume a message from the same influencer topic leading to notification duplication, and you explained consumer grouping as the solution to it. But you might have missed to explain that it is not actually possible, any extra consumers more than one active on a partition sits idle and kafka takes care of it automatically. Viewers might think that as some inherent problem with kafka and they would be required to handle it. Apart from these minor things the video is very good. Thanks for making it!

[@gkcs]

Thank you!

[@NikhilKumar-oy7mx]

I have worked on sqs n sns but this helped me understand that concept Thanks a lot 🎉🎉

[@gkcs]

Cheers!

[@Engineerhoon]

I haven’t received my Amazon delivery yet. What might be the possible reason?

[@sommukherjee2582]

@@Engineerhoon someone in Amazon must have missed a # in Amazon code

[@gkcs]

Sorry are you sure you are on the right video?

[@e.l.5402]

@@gkcs kafka must have lost the order event

[@noah-sheldon]

@@gkcs haha

[@aayushnarayanofficial]

A missing #

[@prakharsinha4145]

awsm explanation bhaiya!... understood every bit!

[@gkcs]

Cheers!

[@ikaaika3155]

Man thanks for sharing this

[@sojanmathew5875]

Hi @gkcs, it looks Kafka partition leader handles both read & write. Write Acknowledgments: The level of acknowledgment (acks) from the leader to the producer depends on the producer configuration: acks=0: The producer does not wait for acknowledgment. acks=1: The producer waits for acknowledgment only from the leader. acks=all: The producer waits for acknowledgment from all in-sync replicas (ISR). [ Read ] Consistency: For strict consistency, consumers can be configured to only read from committed offsets (data replicated to all in-sync replicas). Also newer version of Kafka (2.8 onwards) introduced KRaft (Kafka Raft) as replacement for Zookeeper to simplify operations & system design.

[@gkcs]

Ah, lots of improvements over the 2011 paper I see. Thank you for sharing them!

[@varunv2088]

Awesome! The best kafka internal architecture video

[@bharathraj4196]

Intresting bro looking forward to work in Kafka... Soon i am going to complete my HTML class 🎉

[@satyamjha68]

Amazing video as always! Really informative!

[@gkcs]

Thank you!

[@random_color_lemon]

Nice explanation. Thanks!

[@gkcs]

You are welcome!

[@sudipmandal2497]

Thanks again for a great video.

[@gkcs]

Thank you!

[@kryptu]

very useful , Thanks Gaurav

[@RR-zg5rd]

you are amazing brother!

[@raunaksingh8913]

Very well explained

[@riturajsrivastava9242]

Great thanks Gaurav Sen

[@girishanker3796]

Great video 🤜🤛

[@gkcs]

Thank you :D

[@dhruva_kashyap]

1) Why do you call yourself gkcs 2) These new videos you upload, do you add them to your system design course as well? (I have enrolled and I am trying to figure out if your course is enough, or if I need to watch the videos as well) Great video as always btw 😁

[@gkcs]

1. It stands for General Knowledge Counter Strike. It was my cs handle in college, and it stuck with the KZbin channel :D 2. We typically go into more depth at InterviewReady with the research papers. We dive into the papers as a group during the live zoom sessions. You can see the recordings here: interviewready.io/learn/system-design-course/scaling-memcached-at-facebook/november---memcached-reading Thank you!

[@dhruva_kashyap]

​@@gkcsthank you for taking the time to respond :) I'll surely check it out, all the best!!

[@debasishphukon2314]

I asked the first same question years ago, I initially thought it was General Knowledge Computer Science. @gkcs put up a poll after that If I remember correctly, and then I got to know the real full form

[@cugansteamid6252]

Thanks Gaurav!

[@kartiksachdeva3]

The timing of this video is like it was made for me :P

[@gkcs]

Cheers!

[@prof_as]

thankyou absolute amazing

[@chandrashekark5382]

Well explained, I’m a Frontend developer and heard people talking about Kafka so this helps me to be part of discussion. Also, can you complete the Application cache box at 13:18. It triggered my OCD 😂

[@gkcs]

Hahhaha 😁

[@kaustubhdixit8278]

This is gold...

[@gkcs]

Thank you!

[@aj.arunkumar]

You said kafka uses paxos, but isn't zookeeper atomic broadcast used ?

[@SuperJikks]

I see you have mentioned multiple time that kafka consumer reads from replicas. But I guess this is incorrect, both producer and consumer only interact with leader at any point of time. Can you confirm this? You are right where you mentioned "Consumer consume message only once they are replicated to all the replicas" but still they consume only from leader replicas

[@puneetkumarsingh1484]

At 5:36, why didn't the message no. 4 get replicated to the other partitions after it was received?? That way, when middle kafka broker would receive 5, everything would remain consistent

[@FakeDumbDummy]

that was just a problem scenario for explaining possible data inconsistency which doesn't happen ... if you set acks to leader or min sync replicas to 1 this can still happen

[@Clma333]

This was such an insightful video! I really enjoyed the part about kafka . It reminded me of a video series I recently made on S3 buckets, where I dive deeper into core concept of S3 buckets like replication,storage tiers. If anyone’s interested, feel free to check it out - I’d love to hear your thoughts!

[@gkcs]

Thank you!

[@Clma333]

@@gkcs Thank you sir , I have learned so much from you. trying out to make my own niche

1. What if consumer itself crashes? 2. Will the consumer wait for the write to be replicated across all replicas of the partition?

[@gkcs]

1. It will restart and start pulling messages from it's previous offset. 2. The consumer must wait for the write to be replicated across all partitions (that's the high water mark). It will not find the new messages till they have been replicated.

[@biswaasen]

I can use redis as a quque, also there is rabbitmq, why is it better to use kafka?

[@ashpreetsinghanand7260]

We can also use rabbitMq

[@Goaboyz]

Is the kafka replicas internally that way or the user need to configure it as a system design?

[@gkcs]

They are that way, and can be configured by the admin.

[@saliyamchekka7070]

Why don’t you continue to finish the Kafka course?

[@gkcs]

I have a course at InterviewReady on System Design. Do check it out :D

[@sasiiqoobackup]

can you make videos in dark mode?, it's too bright🐱, the board and animation also white?

[@Md_sadiq_Md]

15:05 money money money 🤑

[@gkcs]

I can feel the cash through the screen :P

[@TheHardikupadhyay87]

is it same like rabbitmq and redis? what is the difference?

[@bytebgmigaming5704]

@@TheHardikupadhyay87 rabit mq is kind of same but it has different qualities and use cases

[@priyalvyas2091]

redis is used for cache and not as message queue

[@swarupcodes]

@@TheHardikupadhyay87 Yes but it’s pull based and not push based like rmq.

[@gpkarnik]

Is it true that zero copy does not work when SSL/TLS encryption is enabled?

[@gkcs]

stackoverflow.com/questions/58651562/how-to-best-address-the-adverse-impact-of-ssl-encryption-on-zero-copy-optimizati

[@sivaguru671]

Zookeeper works like eureka server?

[@gkcs]

Have a look at the third point here: github.com/Netflix/eureka/wiki/FAQ

[@avparadox]

Present Sir

[@gkcs]

:D

[@mehulgoyal6255]

Videos topic is great but i feel there is not a very good explanation to a fresher who watch it very first time

[@himankshu]

what were you doing before speaking for the video ?! 😂

[@aviiiiiiiiiiiiiiiiiiii]

Exactly, I had to pause the video to make sure it wasn't me having a sudden heartbeat rise.😂

[@gkcs]

I was staring creepily at the camera.

[@himankshu]

@@gkcs some design flaw in the system 😂😂

[@shivambansal5572]

you seem like a fan of Suniel Shetty

[@srinivasanranganathan2436]

Gourav Sen is captioned as god of sin 😂

[@agnelodsouza3396]

Your name was spelled as God of Sin in the transcript 😂😂

[@gkcs]

Now it's turned into an Easter Egg :P