Kafka Tutorial CreatingConsumer

Spark Programming and Azure Databricks ILT Master Class by Prashant Kumar Pandey - Fill out the google form for Course inquiry.
forms.gle/Nxk8...
-------------------------------------------------------------------
Data Engineering using is one of the highest-paid jobs of today.
It is going to remain in the top IT skills forever.
Are you in database development, data warehousing, ETL tools, data analysis, SQL, PL/QL development?
I have a well-crafted success path for you.
I will help you get prepared for the data engineer and solution architect role depending on your profile and experience.
We created a course that takes you deep into core data engineering technology and masters it.
If you are a working professional:
1. Aspiring to become a data engineer.
2. Change your career to data engineering.
3. Grow your data engineering career.
4. Get Databricks Spark Certification.
5. Crack the Spark Data Engineering interviews.
ScholarNest is offering a one-stop integrated Learning Path.
The course is open for registration.
The course delivers an example-driven approach and project-based learning.
You will be practicing the skills using MCQ, Coding Exercises, and Capstone Projects.
The course comes with the following integrated services.
1. Technical support and Doubt Clarification
2. Live Project Discussion
3. Resume Building
4. Interview Preparation
5. Mock Interviews
Course Duration: 6 Months
Course Prerequisite: Programming and SQL Knowledge
Target Audience: Working Professionals
Batch start: Registration Started
Fill out the below form for more details and course inquiries.
forms.gle/Nxk8...
--------------------------------------------------------------------------
Learn more at www.scholarnes...
Best place to learn Data engineering, Bigdata, Apache Spark, Databricks, Apache Kafka, Confluent Cloud, AWS Cloud Computing, Azure Cloud, Google Cloud - Self-paced, Instructor-led, Certification courses, and practice tests.
========================================================
SPARK COURSES
-----------------------------
www.scholarnes...
www.scholarnes...
www.scholarnes...
www.scholarnes...
www.scholarnes...
KAFKA COURSES
--------------------------------
www.scholarnes...
www.scholarnes...
www.scholarnes...
AWS CLOUD
------------------------
www.scholarnes...
www.scholarnes...
PYTHON
------------------
www.scholarnes...
========================================
We are also available on the Udemy Platform
Check out the below link for our Courses on Udemy
www.learningjo...
=======================================
You can also find us on Oreilly Learning
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
=========================================
Follow us on Social Media
/ scholarnest
/ scholarnesttechnologies
/ scholarnest
/ scholarnest
github.com/Sch...
github.com/lea...
========================================

[@ScholarNest]

Want to learn more Big Data Technology courses. You can get lifetime access to our courses on the Udemy platform. Visit the below link for Discounts and Coupon Code. www.learningjournal.guru/courses/

[@mpgrewal00]

Love your dedication to educate the community. Creating such presentations that you use to explain consumes time, really appreciate your hardwork and time.

[@abhiganta]

This is the 17th video in this playlist and I still wonder the way you explain each concept. All topics were very clear and the pictorial representation was amazing.

[@vasanthakumarselvaraj6375]

Best Kafka tutorial I have seen. Thank you for the wonderful, neat and clear lecture

[@ScholarNest]

Thanks a lot Vasanth for your feedback. Appreciate you took the time to write back.

[@user-oy4kf5wr8l]

This is the best kafka tutoriel ever !!! Thank you buddy! we love u !

[@sachinsree]

Such excellent tutorial. Your awesome sir!. The way you explain the concepts i liked it very much. It crisp and clear. Thanks!

[@narayanarao2477]

You are best trainer sir even seen as of now

[@rajareddy47444]

Hi,Thank you sir for sharing your knowledge.The way u take the concept and explaining is awesome. After watching your spark videos i got the confidence that i can face interviews with more confidence .Thanks for showing way how to do in real time using GCP. Now i started learning kafka parallel with spark. These videos are in terminal.Can you please explain kafka also in real time with GCP. That will be a great advantage for those who are moving to this ecosystem. Thank you

[@adityasaxena7856]

great explanation Sir, Thank you😊

[@Devans512]

1) 10k producers sending the 50 records(one record at a time) of data for each 3min, can you please suggest me that how many partitions, consumers do i need? 2) What will happen if consumer count is more than Partition count, whether consumers consumes duplicate data( consumes more than once)? 3) what is the RAM consumption, ROM usage? 4) How much time consumers will take to read all the records?

[@ScholarNest]

1) You can decide the number of consumers based on some experiments. I suggest a small POC with the real records and then do some maths. 2) No, additional Consumers will not read any partitions. 3) Memory consumption depends upon your message size, and the number of in-flight messages is the main factors. 4) Depends on many factors. The size of the record and network speed are the major ones.

[@yog2915]

amazing sessions to get ur basics right

[@premkushwaha3111]

awesome tutorial, explained very well and clear all doubts

[@girishkuruba6323]

Excellent video!!!!. Explanation was very clear. Looking forward for more videos.

[@nileshpawar6299]

explained in simple and good way..

[@patilsuhas]

Thanks for good tutorial. Have a question around authentication. How broker authenticate consumers if it is valid? What if someone know the configuration and create his own consumer to pull the data?

[@gunanksood]

Amazing tutorial. Really like the way you explain each and every concept. Thank you

[@kumarvairakkannu360]

Question- Only group name is created and no assignment of how many consumers should be created for a group in order to balance for multiple partitions. Kafka decides numbers of consumers automatically?

[@ScholarNest]

No, Kafka doesn't decide the number of consumers. We have to start Consumers, as many as we want. Just keep the Group Name same for all consumers, and they will share the workload.

[@kumarvairakkannu360]

Thanks a lot. Understand.

[@ashwanishukla744]

How to send data from Kafka consumer in spring boot to Victoria metrics.

[@abhis1083]

very good informative tutorial on kafka

[@ScholarNest]

Thanks

[@aniqatiq731]

Can you describe a clean way of closing the consumer mine keep stuck at the infinite loop

[@sk4162]

how to get latest message enter into topic using java consumer

[@harishtati5667]

Hi sir, i did code same but i am seeing logs updating continuously along with producer messages. I mean when i run consumer its displaying logs continuously without stopping. The logs are like fetching data or uncommitted offsets etc. Please help me why it is coming..

[@praveenppk1]

Excellent way of explanation and Waiting for your Hadoop and hive tutorial as well

[@ScholarNest]

Thanks, It's coming soon, Before the Bahubali 2 :-)

[@karthikkumar12]

lol :)

[@ragavkb2597]

Thanks again for a good tutorial series, At the time of executing consumer.poll(), does it fetches all available messages in the assigned partition at once ? Is there any control to specify how much data(at most) the consumer is interested in receiving in a single poll ? On a related note, suppose that my consumer received 10 messages on a single poll() but failed(consumer died) while processing the 5th message. Is there a way for the broker to know that it has to re-deliver the messages to the newly re-assigned consumer? Appreciate if you could give some insights on such scenarios. Thanks.

[@jay-arabalos4393]

Do you have video tutorial for Junit testing a kafka consumer?

[@SrinivasGadeGPlus]

Isn't we have to pass zookeper details in the consumer... rather passing the brokerlist?

[@pratikgandhi9543]

Which API create coordinator and leader ?

[@prageethjayasinghe709]

Short and sweet!

[@savitaraghuvanshi]

very informative .

[@karthikragunaths]

What happens if there's 5 partitions, and I've created 2 consumer groups, both having 5 consumers? How does the GROUP CO-ORDINATOR balance in such a case? Does it randomly assign the 5 partitions among these 10 consumers? Another question I have is on the message itself -- If a message is too big, is the message split and pushed to different partitions ? or same partition ? [ I'm sorry I couldn't post this in one of the producer related videos ]

[@ScholarNest]

Q1- Each Group will read all 5 partitions. Since there are 5 consumers in each group, they get one partition each. Q2- If the message is too big, you get an exception. It can't be sent.

[@prabhupsgtech]

Best Tutorials for Kafka.. If possible, will you able to post some videos about Mirrormaker?

[@ScholarNest]

Thanks. Your request for the new content is reasonable. I am adding it to my list for next Kafka series. I will start a new set in the near future.

[@prabhupsgtech]

Thanks:)

[@JohnWick-zc5li]

Thanks sir...it is not possible consume and send m sg (s)from the same application for testing purpose

[@rajareddy47444]

I tried to make 2 brokers in GCP. its throwing an error. It says only one boker available for replication faction

[@faisalnkutu3776]

Thank you

[@anushbabukancharla3907]

conumer reads all records from topic higher than current offset or particular number of records? ,if its a particular no of records ,how consumer decides how many records to read from topic can u please explain

[@ScholarNest]

It reads as much as it can within the specified time limit and other limits specified through configurations.

[@ramasubbareddy6147]

good explanation

[@priyankasarkar4527]

Sir how does the consumer get to know that it has finished consuming all records from the producer?Is there any technique for that?

[@ScholarNest]

I think you finish all videos and by the end of tutorial series, you should get the answer.

[@chiragkadam5948]

i want command which create consumers group

[@user-rr3cc3uv2h]

Saying there are 5 partitions in a topic and 4 consumers in a group to consume the topic messages. Now I start a independent new consumer to consume the topic messages, how does the group coordinator to handle this case?

[@ScholarNest]

What do you mean by the independent consumer? If it is part of the new group, it consumes all partitions. If it is not part of any group, then you have no group coordinator assigned.

[@user-rr3cc3uv2h]

you mentioned the concept of 'independentn consumer' at 2:20. I am confused what partition will the independent consumer works on.

[@ScholarNest]

Ok, That's a consumer who is not part of any group, so it has to assign topic and partition to itself using "assign" API. It should take care of everything that a Kafka group management offers. There is an example for such a consumer in "Exactly once processing" video.

[@gowthamjs23]

Hello! Thanks for the great tutorial. How can I find the number of messages in a topic using Java and is there is a way to check if a particular message is in a topic? Please share your thoughts. Thanks!

[@ScholarNest]

You can do all that however you must ask yourself why? Do you want to use a broker as a database to execute queries?

[@HollyJollyTolly]

Thank you sir...

[@linipiusk8965]

can you help me how to copy data (say json format data) consumed from one topic to another topic after data being modified?

[@ScholarNest]

Create a consumer and also a producer, both in same application and you can do it. Join My Udemy course and I am adding an example in the same. It is free for few more days until I finish the full content.

[@mkvjaipur]

Question? what happens if there is a Leader broker fails from where consumer was consuming data?

[@ScholarNest]

A new leader is elected. I have a separate video on Fault tolerance in Kafka. Have you checked that? Hope that answers your doubt.

[@ScholarNest]

However, Do you connect to a Leader? No, Right. When you create a consumer, You just connect to the cluster by supplying any broker address. So, as a consumer, you never know who a leader is? If one Leader goes down, a new leader is elected.

[@Jitendrakumar-mg9gr]

Could u Please help me. I have some dought . Please give a call on that number 9066898340

[@venkataraok]

Where do I need to place property file?, as we are not mentioning the property file path in the code

[@ScholarNest]

current directory.

[@gouraharidas]

Sir, How can I provide a group name if I am starting a console consumer using the kafka script?

[@arka70]

Hi Sir, here group_id is use if we have a consumer group. But is it required if we only use one consumer?

[@ScholarNest]

No. You dont need it.

[@liviucira3611]

What if I do this: props.put("bootstrap.servers", someIpAddress + ":9092"); ? Shouldn't my consumer receive messages from a functioning producer from a host having the IP address = someIpAddress? Is there any problem if many consumers bind to the same port of the producer (i.e. someIpAddress:9092)? For instance I have the following situation: producer on a host, consumer1 on the same host, consumer2 on a different host. Consumer1 receives the messages sent by the producer, consumer2 DOES NOT... the code is identical, I only changed "localhost" with the IP address of the producer (for consumer2).

[@ScholarNest]

I am not sure if I follow your question. However, I read a fundamental misconception. The consumer never directly interact with a producer. There is a Broker in between. The bootstrap server is the Broker address, not the producer. Hope this helps.

[@liviucira3611]

If the consumer is on a different machine than the one of the broker and if i do this in the consumer: props.put("bootstrap.servers", "123.12.12.12:9092"); where 123.12.12.12 is the address of the broker, shouldn't the consumer receive the messages?

[@ScholarNest]

Yes. That's the ideal case. You can't expect consumers to be executing on broker nodes.

[@rishabhsinghal8638]

Sir i have a question is it possible to delete a message from topic once a subscriber have received a message something like subscriber will send an acknowledgement to producer or topic and then message can be deleted.

[@ScholarNest]

No. Subscriber and producer don't know each others. They always talk to broker. They never talk directly.

[@MrJz450]

Hi, I have a question that, can we create kafka consumer for any mysql databases?

[@ScholarNest]

Yes, we can, But for that, we use Kafka Connect.

[@MrJz450]

Thanks for your reply, are you talking about the "Confluent Jdbc" Connector?? Is there any Consumer API(in Java) Available for mysql as a consumer?

[@ScholarNest]

check kafka.apache.org/quickstart#quickstart_kafkaconnect

[@MrJz450]

Thank you so much....I appreciate your Response.

[@antarahazra927]

Can you share the git hub link?

[@ScholarNest]

github.com/LearningJournal/ApacheKafkaTutorials

[@gvbsprasad2179]

.net ?

[@alihossein_shahabi]

how can I download powerpoint of this course?

[@ScholarNest]

PowerPoint is not available. If you are looking for code to copy paste, it is available at the website.

[@alihossein_shahabi]

No. I want to hold a seminar on Kafka in Iran. So I wanted to get help from your good slides

[@ScholarNest]

Unfortunately, I haven't preserved those slides. You might want to take the screenshots of full-screen videos.

[@alihossein_shahabi]

ok.thanks