Kafka Consumer Offsets Explained

[@MohitSaini-tv8go]

@Aman Bro your kafka series helped me to crack difficult interviews in difficult situations ..You helped me to clear my concepts!!!! Please make more videos ... Appreciated !!!!!!

[@elighteloy1810]

Thank you so much from your video. You make me understand clearly in Kafka.

[@raphy78626]

Really nice I have seen many videos none were clear on these details

[@SanjeevKumar-qv4kc]

Nice video about consumer offset👍👍

[@nikhil-zz6mr]

can you please explain at 8:32 how can a commit be at an offset 10 when the processing is happening for the messages at the earlier offsets?

[@MohitSaini-tv8go]

Good one Aman !! Thanks

[@swastikbhomkar7296]

Thanks! Understood more deeply now.

[@TechWithAmanYadav]

Glad it was helpful!

[@ayushgarg5929]

Please make such a playlist on SQS Also

[@oavilex]

Nice explanation

[@lohithreddy9875]

Hi Your videos are so informative and easy to understand.Thanks a lot. Can you please make a video of scaling tips and how to reduce load on consumers using configs.

[@manugoel467]

Great video, in a concise manner. Answered most of my doubts. Best of Luck Aman!!

[@seenimurugan]

good one. Thank you.

[@sudippandit1]

Hi, how do we handle the duplicates values in Kafka?

[@rohitkumarnode4839]

Hello Tech Aman, you said in case of manual commit , you can commit the messages individually once particular message processed, but as we know that kafka always keeps the status of largest offset committes, suppose mesage with 2,3 offset hsn't processed and message with offset 4 gets processed we commit the offset 4, in that case 2,3 offset will also be considered as committed. how we can remove this issue ?

[@TechWithAmanYadav]

Hi Rohit, You can commit any offset. But in usecase you have mentioned why 2,3 are not processed? Since a single consumer will be processing message in order. And if that is the case then you would have already processed 2,3. Other if these are in different partition then that dosnt bother this partition offsets as offsets are committed per partition basis.

[@RohitKumar-bl2eg]

@@TechWithAmanYadav I am processing messages in parallel not in particular order , so I guess committing individually won’t work correctly. Any thoughts ?

[@InderjeetSingh007]

Any code repo for implementation of above?

[@TechWithAmanYadav]

Not right now

[@rohitkumarnode4839]

Also one more doubt i have, how we can consume same message twice upon two successive consumer poll, i have this requirement where if previous messages didn't get processed and i have not committed i want to receive those same messages again, but it seems to consume next batch of messages because somehow kafka remembers that which offsets was consumed during previous poll. In Simple words how can we reset the in memory offsets ?

[@TechWithAmanYadav]

if you don't commit the message, then in next poll the same message will be consumed again. And remember to turn off auto commit option.

[@RohitKumar-bl2eg]

@@TechWithAmanYadav I tried the same but it’s not happening, I am using node-rdkafka npm library.

[@satishkuppam8008]

I am trying to override auto.offset.reset in cosumer.properties and connect-standalone.properties file but it is not overriding. Can you please tell me how to fix this and i have added the connector.client.config.override.policy=All in connect-standalone file.

[@saurav0777]

I have tried saving the offset in kafka itself in my spark streaming app using commitAsync API. However it is not syncing on time or immediately in kafka internal topic __consume_offset because of that spark is processing duplicates records whenever it restarted even though doing graceful restart . Could you please give some pointers to fix ?

[@TechWithAmanYadav]

You can acknowledge in sync and send acknowledgement after processing each message.

[@srimayeeponuganti775]

@@TechWithAmanYadav any reasons for consumer processing same message twice?

[@TechWithAmanYadav]

Well there can multiple scenarios. There might be some db crash and you need to process messages again.

[@sudheerkumaratru9418]

I am not able to listen for the particular topic @kafkalistener Is not working

[@neha6000]

Implementation kaha hain..?

[@sudheerkumaratru9418]

Hii

[@ManishTiwari-or8zt]

Not good content... It can be better way to explain...

[@sudippandit1]

Hi, how do we handle the duplicates values in Kafka?

[@TechWithAmanYadav]

are you pushing same value twice?

[@sudippandit1]

Yes due to the network issues, streaming job got failed and restarted by itself and same row values got populated twice