Kafka Tutorial Rebalance Listener

[@ScholarNest]

Want to learn more Big Data Technology courses. You can get lifetime access to our courses on the Udemy platform. Visit the below link for Discounts and Coupon Code. www.learningjournal.guru/courses/

[@nemosourav]

I want to extend my sincere thanks for this amazing course. I have been following your kafka playlist and being a kafka-newbie, this course has been amazing. Again, many thanks for your efforts.

[@palasuresh1987]

Thanks a lot for creating this playlist and making most of the people lives easy to understand the Kafka. I would say it is one of the finest explanations to understand the kafka starting from the scratch.

[@vikramvarshney3224]

I am working on Kafka 0.9.0 last 1 year and facing the problem of rebalancing by the coordinator while consumer process is no longer be running. While I restart consumer process it is taking rebalancing always. Now I have exactly understand how to commit offsets properly. Thanks a lot for sharing the valuable knowledge.👍🏻

[@StrongbowJava]

Thank you very much for your perfect explanation, which helps me a lot to understand the complex topic of partition assignment and revocation. I like your slow and clear voice, so that I could follow you without a problem. Keep up the good work.

[@anumsheraz4625]

Best tutorial I found so far. Thank you so much Sir for sharing your knowledge.

[@niftymiller6057]

Amazing videos, way you present to audience, the flow is unbeleivable i cannot thank you more

[@peterabiodunokusolubo1541]

This is a fantastic tutorial, I've been looking for this explanation for some time. Thanks

[@kumarvairakkannu360]

This is fantastic, loved your presentation! Also this video cleared my previous doubt...Thanks a million!!!

[@gagangupta1255]

Simply awesome video explaining the rebalancing

[@glpathy]

Thank you sir for a practical explanation with a problem statement and a resolution.

[@nooruskhan5200]

This is really a nice tutorial to start with kafka, thanks for all your effort

[@cellisisimo]

Very clear and concise explanation of Kafka!! I cannot wait for other tutorials. Thanks!!

[@ScholarNest]

Thanks a lot for the feedback, I am working and committed to at least one every week.

[@UnicornTwichu]

Sir you have perfectly represented and explained all.

[@TheMcallist1]

Brilliant explanation - thanks

[@nodeflowTrader]

good job in explaining kafka in this way - straight to the point I like it :)

[@rajareddy47444]

Hi,Thank you sir for sharing your knowledge.The way u take the concept and explaining is awesome. After watching your spark videos i got the confidence that i can face interviews with more confidence .Thanks for showing way how to do in real time using GCP. Now i started learning kafka parallel with spark. These videos are in terminal.Can you please explain kafka also in real time with GCP. That will be a great advantage for those who are moving to this ecosystem. Thank you

[@venkatakrishna5222]

Hi Sir, Thank you so much for the excellent explanation. I have few doubts. 1. Can rebalance possible in single partion, single consumer scenario. If possible do we need to implement rebalance listener for this scenario. 2. Can we statically bind that partition to that consumer. 3. Can we disable rebalance listener. Thank you very much sir 🙏

[@bhanud1806]

Your Complete tutorial is very good and clear to follow. Thanks a lot for sharing us this info :)

[@mohanbabubanavaram5211]

Excellent Explanation

[@DavidFraserable]

These vids are amazing. You rock!

[@venkatakrishna5222]

Hi Sir, your Kafka teachings are fantastic. Can you please teach on Spring Kafka. Thank you Sir 🙏

[@vineettalashi]

Great video

[@PA-vf5st]

Sir Fantastic tutorial and superbly explained....One question Can we not commit partition passed in onPartitionsRevoked method to avoid commtfailedexception..?

[@naveenkumarmurugan1962]

Great work sir.... God bless you..

[@theashwin007]

Superb! I think this is THE BEST tutorials on KAFKA. When I searched few months back, there were no this clear recordings. Thanks for getting this to the community. I have one question. Can we configure how many records Consumer can read in one poll ? I mean say there are 1000 records at the broker level and when you say number '20' records returns in your example, I hope that is a max-configured one? Like several producer configs, can you please explain different kind of polls at the consumer side?

[@ScholarNest]

We try to control max number of records by putting various limits like max.poll.records and passing a timeout parameter to poll method. The poll method tries to get you as many as possible within those limits.

[@virenmathur]

Hi, This is a very nice series for Kafka.Probably the best. Thanks for putting it together. Wanted to get clarity - The commit in the consumer code is commented out. Is it because you want to show how rebalancing works? So had two questions for you 1. How the commit will happen when next loop comes around. Would you uncomment the consumer.commitSync(.......) ? 2. Do I need to write additional code in the rebalance listener to clear out the offset map. My understanding would be that the Map would get overwritten. Please suggest. Thanks Viren

[@chaitanyatanwar8151]

Thank you!

[@deepaksoundappan3244]

Greate work Sir!!!!. This is great material is saving Tons of money to people who will have to other wise take stupi course in Edureka or other sites :)

[@TejpalSingh-qe5ct]

@Learning Journal Shouldn't we use ConcurrentHashMap as multiple consumers will try to update the map simultaneously?

[@SatishKumar-ix9mz]

Thanks a lot !! Excellent tutorial ..

[@avismg222]

Very good explanation..awesome..

[@ScholarNest]

Thanks a lot for your encouragement.

[@adershrp]

Thank you.

[@imochurad]

Hi, thanks for the great video. I have the following question here: you are giving an example of how to make processing of a single message and commit an offset as a single atomic operation. This is easily achieved in your case since you are storing data in the same database. So you are utilizing JDBC transaction. What should I do if I have to consume a message and then I am supposed to POST it to the external API instead of saving it in the database? If my POST to the API was successful and my consumer crashed between storing offset information, I think my message might be processed twice.

[@vinayyelleti2714]

Can u pls explain how can we achieve the same when @kafkalistner is used

[@rickytik-devops]

so your saying the kafka can either be a producer/sender or consumer/reciever?

[@banam3540]

Hi, I have a situation. I have multiple consumers on same topic. One of consumer received 10 events to process. After processing 6 event all of the sudden consumer goes down. I am using auto commit. Now my problem is what is status of 4 items which is in consumer queue. Waht happens to current and commit offset. How can stop data loss situation here. Please help me. For ur inform we using Azure Function Kafka Trigger.

[@Satyakam85]

Thanks Mr. LJ (unfortunately I haven't figured out your name yet :( ) for the great tutorials on Kafka. I have a query which is bothering me for a while. Is it a good practice to use the following for processing of records: 1. Use JMS to which Consumer can drop messages (records from Kafka Broker) for an actual end application to process gracefully. That way one could atleast commit offsets without the need to wait long ? 2. Use another Kafka Broker (may be a separate instance for processing of records). So basically pull stream of data from a broker and push to another broker. I know sounds crazy but is it used anywhere ?

[@ScholarNest]

Let me understand your question.1. You want to use JMS as a buffer because you think processing message will take a long time and we have to wait for that before we commit. Are you sure? I guess, it will complicate the solution and I don't see a reason. 2. I guess you mean pull from one topic, process it and push to another topic ( you didn't mean - from one broker to another broker). I wonder, why would you need another Kafka Cluster.

[@Satyakam85]

Thanks sir. I guess I get when you say JMS would complicate the solution as we are ideally going from distributed messaging to single point messaging. As for point 2, it was just a wild thought. However do we have topic to topic routing in Kafka ?

[@vaibhavbacchav2674]

One small doubt sir, Is re-balancing activity local to the consumer group? Ex. Suppose we have 2 consumer gropus CG1 & CG2 and If consumer gets added/removed to/from CG1, it will trigger re-balancing only for CG1 or it will trigger re-balancing for CG2 also PLEASE REPLY ASAP, thanks alot GREAT TUTORIALS, hatsoff!

[@ScholarNest]

Yes, You got it correct.

[@vaibhavbacchav2674]

@@ScholarNest thanks alot prashant sir

[@ashokmacherla4573]

How to calculate number partitions are required for topic??

[@UnstoppableAadhya]

I am writing a service to retrieve past messages of a topic and trying to show.Can you please help me which approach is better choice for this.

[@hadoopworld35]

Hi Sir.. Thanks for this tutorial. I have one doubt , How we will do it if we dont know the partition no... what would be the addoffset() here ..can we write TopicPartition only with topic argument

[@hadoopworld35]

Ok...So We can not create topic without partitions

[@aravinthanm3183]

Sir how to clear offset value. pls help me

[@vaibhavbacchav2674]

And this re-balancing won't happen if I add a producer which starts writing to the same topic from which a consumer group is reading (no addition/subtraction of consumer when producer got added) ?

[@ScholarNest]

Rebalancing has nothing to do with the producer.

[@vboilay]

your tuto are excellent. let me know how I can reward you . tipee?

[@ScholarNest]

Your appreciation is my reward.

[@rikuntri]

For starting KAFKA we need to start zookeeper.but in cloudera VM i think there is zookeeper instance already running .SO can anyone tell me how to check through VM terminal whether the default instance is runnig or not ? Also if we start a zookeeper service we can see the logger screen but how to check it for cloudera VM.

[@ragavkb2597]

Once onPartitionsRevoked() call back is received, I understand we are trying to commit the offset. I think we should also stop processing the remaining messages, am i correct ? Suppose during the last poll() we received 100 messages and at the time of processing 51th message we received onPartitionsRevoked() callback and we would call commitSync that we processed 50 messages so far. Messages from 51 through 100 may get processed by the newly assigned consumer, won't that end up in duplicate processing of records? Thanks

at 8.18, in addOffset method, at the end, you write new OffsetAndMetadata(offset,"commit"); what does it mean? why you put a string "commit"?

[@ScholarNest]

That's metadata. It could be any string,. You can use any non-null string instead of Commit.

And what's the purpose? You're putting in the hashmap with key topicpartition a value of Offset and Metadata, right? but I don't understand where you use it and what could be useful for. Thank you :)