Internals of Kafka Consumer | Offsets in Kafka | Apache Kafka Tutorial in Hindi | Video 10

This video explains how kafka consumer works and a detailed overview of offsets and their types.
You will get answers to below questions in this video
what is kafka consumer
what is kafka consumer lag
what is kafka consumer group id
how kafka consumer read from partition
how kafka consumer works internally
how kafka consumer poll work
how kafka consumer maintains offset
where is kafka consumer offset stored
Types of offsets explained
Log-end offsets
Current offsets
Committed offsets
Find below the video on "Internals of Kafka Producer" in Kafka.
• Internals of Kafka Pro...
Find below the video on "Topic creation, run console Producers, run console Consumer" in Kafka.
• Topic Creation | Run C...
Find below the video on how to setup Zookeeper cluster with 3 zookeeper node on local machine.
• Zookeeper Cluster Setu...
Find below the video on how to setup Kafka cluster with 3 broker on local machine.
• Kafka Cluster Setup | ...
Find below the link to Kafka tutorial series Playlist.
• Apache Kafka Tutorial ...
Find below the official documentation of Apache Kafka.
kafka.apache.o...
I graduated from IIT BHU Varanasi and currently work as a Software engineer with a MNC company.
MIC :- amzn.to/34Z12rr
TRIPOD :- amzn.to/3FxFI91
LIGHT STAND :- amzn.to/322VXNr
Green Curtain :- amzn.to/3KdKbB6
Connect with me on LinkedIn.
/ brijeshgupta96
Connect with me on FaceBook.
/ brijesh.iitbhu
Connect with me on Instagram.
/ brijeshgupta__

[@vaishnaviboje]

best at 1.75x speed, neatly expalined 👏👏

[@sumitkumar-pt2nn]

Bhai dil jeet liye ho aap. Maa kasab kafka ka sab jhol hi clear ho gya mera

[@irshankhan7158]

zol 😂

[@Amitloveism]

Good video covering all necessary topics 👍

[@vinayyelleti2714]

Please explain 2 ways of manual commit through spring code

[@BrijeshGupta]

Sure will do that in later videos

[@fossmaniac2710]

Hey Thanks for this awsm vido. Plz can you explain . What happen when consumer(down) not able to recvd msg , how we can recover tht msg. How and where producer work internally(what kind of mechanism producer have internally to find specific consumer & snd a msg to thm) & How consumer consume msg from producer like wht kind of acknowledge in between when producer recvd(push) msg from client & how consumer knows to find tht msg (internally wrk). Thanks.... Highly appreciated.

[@BrijeshGupta]

all of these are explained in later videos

[@amanvikram04]

Well explained.

[@arkaimps]

Really impressed how you are explaining the things in your lectures. Highly appreciated.. A question, when you say that Committed offset can never be larger than Current offset, which is understandable when we are dealing with one Partition. But if Consumer is polling from a TOPIC level and its records are spread across 3 partitions and as Offset is tightly associated with Partition, how the values are saved in case of Current Offset and Committed Offset. Is it something like Partition--? I am bit confused when data/message is scattered across partitions.

[@BrijeshGupta]

Thanks for the appreciation Arkajyoti. 😊 The answer to your question is "Kafka stores current offset and committed offset for each consumer group for each partition its pulling messages from."

[@arkaimps]

@@BrijeshGupta thanks for coming back.. one of the video regarding the rebalancing helped also , to understand the transaction...

[@arvindagrawal6937]

in auto commit, suppose message {0,1,2,3,4} sent to consumer. so here consumer will consume the message in same order {0,1,2,3,4} or may consume in any random order and will return offset value to broker in random order executed offset value?

[@BrijeshGupta]

Commit has no relation with ordering of message.

[@arvindagrawal6937]

@@BrijeshGupta just for understanding I am asking. If consumer return offset value which will be considered as committed-offset on a partition. If consumer consume the message in random then it will return the offset which can be random and in this case random offset will be maintained in committed offset?

[@BrijeshGupta]

Ordering is maintained per partition level. Consumer will never recieve random messages from a partition.

[@kewalsonone6249]

Hi bro, I'm getting error on "asynchronous auto commit of offsets failed offset commit cannot be completed since the consumer is not part of the group wil continue to join group". What is solution for this ?? Once msg on Kafka it consumes properly .

[@subhendurana6457]

Awesome!

[@BrijeshGupta]

Thank you! Cheers!

[@er_vikramkumar]

One question regarding committed offset..let's suppose we have 5 consumer In consumer group and as u said controller maintain the committed offset details at consumergroup level... My doubt is there could be a case in consumergrup we had 5 consumer and anyone consumer still not consumed any record and other consumed. What happened in that case will the committed offset will updated at controller end or it will wait till the all consumer in consumergrup consumed all records till the last poll.

[@BrijeshGupta]

each consumer has a privilege to poll independently as well as commit independently. So once a consumer has finished processing the message it can commit

[@piyushpokharna999]

If auto-commit is enabled every 5 sec, and if consumer crashes, then how auto-commit signal will be raised from consumer to Kafka? so logically there should not be any commit and but once the consumer restarts , then after 5 seconds will it auto-commit all the the earlier offsets?

[@suryajoshi9677]

Hello Sir, I have a requirement to stop sending/receiving the message in Kafka topic. Is there a way I can send the notification to kafka topic to start at given point of time and resume it?

[@BrijeshGupta]

Hi Surya, Message are Polled by consumer, not pushed by kafka cluster. So in order to stop receiving messages, u might want to stop the clients.

[@roshankumargupta46]

Hi Bro, Thanks for this wonderful video. Are you also going to cover further topics with python?

[@BrijeshGupta]

Hi Roshan, Thanks for the appreciation. For now I'll be covering the topics Java, later I am planning to upload a quick kafka project with python as well.

[@neha6000]

Particle implementation kaha hain 😢