Apache Kafka in 15 minutes

[@gkcs]

Thank you for watching. Also, NEW EPISODE: interviewready.io/learn/system-design-course/building-an-ecommerce-app-1-to-1m/7-the-case-of-the-expensive-cache

[@kidkool27]

Wow I was just out of college when I first found your channel. I’m a fresh senior now preparing for interviews, great video!

[@gkcs]

Glad to read this, cheers!

[@ashutoshsamantaray6596]

zookeeper is no longer used in the latest version of kafka. The newer version is Kraft using consensus protocol among brokers. Makes it more scalable

[@gkcs]

Yes, they use RAFT now 😁

[@TheCodingChamelion]

Tomorrow is exam and this video was saviour for the kafka topic

[@SaiBharatBatchu]

It’s a great video. It covered the features of kafka well. I feel the way you explain a problem and explain how to overcome it might not be obvious to the listeners that kafka automatically takes care of these. For eg, you said multiple consumers might consume a message from the same influencer topic leading to notification duplication, and you explained consumer grouping as the solution to it. But you might have missed to explain that it is not actually possible, any extra consumers more than one active on a partition sits idle and kafka takes care of it automatically. Viewers might think that as some inherent problem with kafka and they would be required to handle it. Apart from these minor things the video is very good. Thanks for making it!

[@gkcs]

Thank you!

[@tayyabachowdary1007]

Have been following you since long and i am so impressed with the consistency and humility. Yoir explanations are simple and you seem like a college mate we are getting to learn from.thanks for all the free content

[@gkcs]

Cheers!

[@NikhilKumar-oy7mx]

I have worked on sqs n sns but this helped me understand that concept Thanks a lot 🎉🎉

[@gkcs]

Cheers!

[@bharathraj4196]

Intresting bro looking forward to work in Kafka... Soon i am going to complete my HTML class 🎉

[@Engineerhoon]

I haven’t received my Amazon delivery yet. What might be the possible reason?

[@sommukherjee2582]

@@Engineerhoon someone in Amazon must have missed a # in Amazon code

[@gkcs]

Sorry are you sure you are on the right video?

[@e.l.5402]

@@gkcs kafka must have lost the order event

[@noah-sheldon]

@@gkcs haha

[@aayushnarayanofficial]

A missing #

[@sojanmathew5875]

Hi @gkcs, it looks Kafka partition leader handles both read & write. Write Acknowledgments: The level of acknowledgment (acks) from the leader to the producer depends on the producer configuration: acks=0: The producer does not wait for acknowledgment. acks=1: The producer waits for acknowledgment only from the leader. acks=all: The producer waits for acknowledgment from all in-sync replicas (ISR). [ Read ] Consistency: For strict consistency, consumers can be configured to only read from committed offsets (data replicated to all in-sync replicas). Also newer version of Kafka (2.8 onwards) introduced KRaft (Kafka Raft) as replacement for Zookeeper to simplify operations & system design.

[@gkcs]

Ah, lots of improvements over the 2011 paper I see. Thank you for sharing them!

[@raunaksingh8913]

Very well explained

[@RR-zg5rd]

you are amazing brother!

[@kryptu]

very useful , Thanks Gaurav

[@random_color_lemon]

Nice explanation. Thanks!

[@gkcs]

You are welcome!

[@sudipmandal2497]

Thanks again for a great video.

[@gkcs]

Thank you!

[@satyamjha68]

Amazing video as always! Really informative!

[@gkcs]

Thank you!

[@dhruva_kashyap]

1) Why do you call yourself gkcs 2) These new videos you upload, do you add them to your system design course as well? (I have enrolled and I am trying to figure out if your course is enough, or if I need to watch the videos as well) Great video as always btw 😁

[@gkcs]

1. It stands for General Knowledge Counter Strike. It was my cs handle in college, and it stuck with the KZbin channel :D 2. We typically go into more depth at InterviewReady with the research papers. We dive into the papers as a group during the live zoom sessions. You can see the recordings here: interviewready.io/learn/system-design-course/scaling-memcached-at-facebook/november---memcached-reading Thank you!

[@dhruva_kashyap]

​@@gkcsthank you for taking the time to respond :) I'll surely check it out, all the best!!

[@debasishphukon2314]

I asked the first same question years ago, I initially thought it was General Knowledge Computer Science. @gkcs put up a poll after that If I remember correctly, and then I got to know the real full form

[@cugansteamid6252]

Thanks Gaurav!

[@kartiksachdeva3]

The timing of this video is like it was made for me :P

[@gkcs]

Cheers!

[@Clma333]

This was such an insightful video! I really enjoyed the part about kafka . It reminded me of a video series I recently made on S3 buckets, where I dive deeper into core concept of S3 buckets like replication,storage tiers. If anyone’s interested, feel free to check it out - I’d love to hear your thoughts!

[@gkcs]

Thank you!

[@Clma333]

@@gkcs Thank you sir , I have learned so much from you. trying out to make my own niche

[@prof_as]

thankyou absolute amazing

[@kaustubhdixit8278]

This is gold...

[@gkcs]

Thank you!

[@Md_sadiq_Md]

15:05 money money money 🤑

[@gkcs]

I can feel the cash through the screen :P

[@TheHardikupadhyay87]

is it same like rabbitmq and redis? what is the difference?

[@ashpreetsinghanand7260]

We can also use rabbitMq

[@SuperJikks]

I see you have mentioned multiple time that kafka consumer reads from replicas. But I guess this is incorrect, both producer and consumer only interact with leader at any point of time. Can you confirm this? You are right where you mentioned "Consumer consume message only once they are replicated to all the replicas" but still they consume only from leader replicas

1. What if consumer itself crashes? 2. Will the consumer wait for the write to be replicated across all replicas of the partition?

[@gkcs]

1. It will restart and start pulling messages from it's previous offset. 2. The consumer must wait for the write to be replicated across all partitions (that's the high water mark). It will not find the new messages till they have been replicated.

[@Goaboyz]

Is the kafka replicas internally that way or the user need to configure it as a system design?

[@gkcs]

They are that way, and can be configured by the admin.

[@gpkarnik]

Is it true that zero copy does not work when SSL/TLS encryption is enabled?

[@gkcs]

stackoverflow.com/questions/58651562/how-to-best-address-the-adverse-impact-of-ssl-encryption-on-zero-copy-optimizati

[@dnbdsj]

can i use this on my 500mb vps for my todo application with 2 user.

[@siddharthshankarpaul]

U can. But you should not send the 500mb file to Kafka, rather the metadata like location of the file.

[@gkcs]

2 users is a lot. This is a job for ElasticSearch.

[@sivaguru671]

Zookeeper works like eureka server?

[@gkcs]

Have a look at the third point here: github.com/Netflix/eureka/wiki/FAQ

[@Aditya_Vyas]

Present Sir

[@gkcs]

:D

[@himankshu]

what were you doing before speaking for the video ?! 😂

[@aviiiiiiiiiiiiiiiiiiii]

Exactly, I had to pause the video to make sure it wasn't me having a sudden heartbeat rise.😂

[@gkcs]

I was staring creepily at the camera.

[@himankshu]

@@gkcs some design flaw in the system 😂😂