Kafka Tutorial - Producer Configs

Spark Programming and Azure Databricks ILT Master Class by Prashant Kumar Pandey - Fill out the google form for Course inquiry.
forms.gle/Nxk8dQUPq4o4XsA47
-------------------------------------------------------------------
Data Engineering using is one of the highest-paid jobs of today.
It is going to remain in the top IT skills forever.
Are you in database development, data warehousing, ETL tools, data analysis, SQL, PL/QL development?
I have a well-crafted success path for you.
I will help you get prepared for the data engineer and solution architect role depending on your profile and experience.
We created a course that takes you deep into core data engineering technology and masters it.
If you are a working professional:
1. Aspiring to become a data engineer.
2. Change your career to data engineering.
3. Grow your data engineering career.
4. Get Databricks Spark Certification.
5. Crack the Spark Data Engineering interviews.
ScholarNest is offering a one-stop integrated Learning Path.
The course is open for registration.
The course delivers an example-driven approach and project-based learning.
You will be practicing the skills using MCQ, Coding Exercises, and Capstone Projects.
The course comes with the following integrated services.
1. Technical support and Doubt Clarification
2. Live Project Discussion
3. Resume Building
4. Interview Preparation
5. Mock Interviews
Course Duration: 6 Months
Course Prerequisite: Programming and SQL Knowledge
Target Audience: Working Professionals
Batch start: Registration Started
Fill out the below form for more details and course inquiries.
forms.gle/Nxk8dQUPq4o4XsA47
--------------------------------------------------------------------------
Learn more at www.scholarnest.com/
Best place to learn Data engineering, Bigdata, Apache Spark, Databricks, Apache Kafka, Confluent Cloud, AWS Cloud Computing, Azure Cloud, Google Cloud - Self-paced, Instructor-led, Certification courses, and practice tests.
========================================================
SPARK COURSES
-----------------------------
www.scholarnest.com/courses/s...
www.scholarnest.com/courses/s...
www.scholarnest.com/courses/s...
www.scholarnest.com/courses/s...
www.scholarnest.com/courses/d...
KAFKA COURSES
--------------------------------
www.scholarnest.com/courses/a...
www.scholarnest.com/courses/k...
www.scholarnest.com/courses/s...
AWS CLOUD
------------------------
www.scholarnest.com/courses/a...
www.scholarnest.com/courses/a...
PYTHON
------------------
www.scholarnest.com/courses/p...
========================================
We are also available on the Udemy Platform
Check out the below link for our Courses on Udemy
www.learningjournal.guru/cour...
=======================================
You can also find us on Oreilly Learning
www.oreilly.com/library/view/...
www.oreilly.com/videos/apache...
www.oreilly.com/videos/kafka-...
www.oreilly.com/videos/spark-...
www.oreilly.com/videos/spark-...
www.oreilly.com/videos/apache...
www.oreilly.com/videos/real-t...
www.oreilly.com/videos/real-t...
=========================================
Follow us on Social Media
/ scholarnest
/ scholarnesttechnologies
/ scholarnest
/ scholarnest
github.com/ScholarNest
github.com/learningJournal/
========================================

[@ScholarNest]

Want to learn more Big Data Technology courses. You can get lifetime access to our courses on the Udemy platform. Visit the below link for Discounts and Coupon Code. www.learningjournal.guru/courses/

[@suryabh7387]

This is an awesome tutorial.. I got entire concept to start a project in couple of hours. Thank you!!

[@KoushikPaulliveandletlive]

No paid course as such good and brief explanation.

[@nilaytiw]

The way you are explaining it is very helpful to understand such a complex thing

[@mahammadshoyab9717]

soory to ask so many questions,your way of explanation is simply awesome ,if it goes for all bigdata skills like hive,hbase,sqoop,pig,spark sql, etc.......it will be very helpful for alot of people.

[@mahanteshangadi6738]

Really ... your presentation is so simple to understand

[@princesanghi6464]

Hi Thanks for such a nice explanation. Can we configure the Producer Configuration topic wise?

[@durgadeep4988]

Does Kafka guarantee messaging ordering for messages in a specified batch.size ?

[@shekharpandey166]

Sir, in my college project, I am fetching my data from mysql using logstash to elasticsearch. But now I am using in between kafka. So can you kindly tell me how to transfer data from logstash to kakfa. and later I can fetch from that to elastic.

[@chaitanyatanwar8151]

Thank you!

[@swasavlogs]

Hi, I've recently started working on spark, I want to post all messages in one shot to the Kafka but I've challenge over here I've about 1M message are there to post but it can able to post few before timeout, Can you help me with POST all or not all before retry What will be best configs for me, 1. No need of order guaranteed 2. Should not post duplicate messages on retry

[@krushnaswain8887]

On failure of asynchronous call, should I write retry logic inside onFailure method ? Or, it will be taken care by producer retry config parameter?

[@MrPavel0]

what abaut wab sarrvar?

[@zeinabashjaei3302]

Thanks for this excellent collection. I have a question about offsets.topic.replication.factor parameter that does not make sense for me. Could you please provide some explanation. Also how it differs from in-sync replica (ISR). Thank you

[@ScholarNest]

Replication factor tells how many copies do you want. ISR tells who has the latest copy same as the topic leader.

[@archanarajput5160]

can you explain linger.ms property?

[@ramasubbareddy6147]

I have basic question like in a given kafka cluster can have one leader always or many leader ?

[@ScholarNest]

One leader. However, I think you are missing one important point. It is not the leader of the cluster. It is the leader of the partition. So in fact there are many leaders in a cluster, one for each partition.

[@TheNayanava]

I thought, the problem of ordering could also be solved, if we send the timestamp as a parameter to the ProducerRecord object. Please correct me if I am wrong

[@sumitjha392]

With your approach, consumer would have to pull entire data from the queue and then sort (based on timestamp parameter) at its end. But this is not feasible. We would not know the range in which the message are not ordered, pull that entire range and then sort at consumer end. Anyway that's a design decision, in theory a consumer can *insert* message at the appropriate position as it pulls message from Kafka (which again as I said depends on problem you are solving)

[@MonojitBarua]

hello Sir, your explanation is very helpful. one question, if acks=0 and at the same time if RecordMetadata r = producer.send().get(); gets called then will r going to get the record metadata informations as acks=0 ?

[@KoushikPaulliveandletlive]

I believe it will not.

[@charlsjoseph]

if you set the acks config to 0, you would not get the acks and wont call any call backs in case of async and future.get() in case of sync calls. But when i set this to 0, both cases it invokes the callbacks(async) and future.gets called. what does this mean ?

[@ScholarNest]

+Charls Joseph did I say that? or you read it somewhere?

[@charlsjoseph]

my understanding is that if you set the ack to 0, producer wouldnt get the acks back from kafka broker. but when i tried with ack = 0 for async, the callback is getting called. ideally this is not supposed to happen if the broker is not acknowledging the message, correct ? I didnt get the actual use case of ack=0

[@ScholarNest]

+Charls Joseph Well, your understanding is incorrect. Watch the video once again. A bit carefully this time. acks controls the acknowledgement from followers for replication, not the acknowledgement from broker to producer.

[@LivenLove]

@@ScholarNest Sir at 3:42, you mentioned that producer get acknowledgement from broker as metadata or exception.

[@rajasekharchittiprolu797]

Hi sir please correct the video if what u said in above message is correct

[@mkvjaipur]

max inflight is still not clear, could you please elaborate it. why its limited upto 5

[@ScholarNest]

They consume memory. You can increase the number if you have enough memory to buffer more messages.

[@pramodsripada2380]

Learning Journal, by memory do you mean the producer should keep a buffer of all the messages until it all the in-flight requests are resolved?

[@psangawar7130]

Also once the max is reached for inflight, the other messages become synchronous? This was not too clearly explained I think...Appreciate your patience in answering and the awesome tutorials...

[@ashick91]

where can we set these producer configs?

[@aijaz4441]

Since the Bootstrap, key, value and the custom serializer were sent as Properties object, I am assuming these configurations are also set the same way.

[@saisantoshkalla113]

For ordering guarantee you gave two option 1. Synchronous send 2. max.in.flight.request.per.connection=1 In the example, you explained that if the first batch of 5 messages failed then the second batch of 5 messages will be tried and if it succeeds then there will be problem in ordering. In the similar way, when the max in flight requests per connection is set to 1 (async send) and if it fails then the second message will be tried and if it succeeds then also there will be order issues. Am i correct? If i'm correct please explain how this is a solution for ordering guarentee. correct me if am missing something here.

[@LivenLove]

I have the same question.. could you please answer sir??

[@0equalsTo1]

max.in.flight.requests.per.connection means the producer can not (async) send more records than this value(default 5 in the current version) until unless at least one of those requests has been acknowledged. max.in.flight.requests.per.connection=1 means after sendind 1 record producer have to wait for that response.

[@vivekpratapsingh9957]

can you do something about the sound and picture at the starting of the video? doesn't give a good vibe and put it like a childish content though it's not.

[@ThePentanol]

Poor explanation of the third parameter