buffer.memory , max.block.ms & Producer IO Thread in kafka

In this video , 2 very important properties related to kafka producer are covered with in-depth intuition -- buffer.memory & max.block.ms
buffer_memory is the total bytes of memory the producer should use to buffer records waiting to be sent to the server. If records are sent faster than they can be delivered to the server the producer will block up to max_block_ms.
Prerequisite:
--------------------
Install and run Apache Kafka & integration with Python using kafka-python
• Install and run Apache...
Multiple Producer & Multiple Consumer in a Kafka Topic
• Multiple Producer & Mu...
Intuition on Log files in kafka & Kafka Brokers
• Intuition on Log files...
Broker Cluster and Zookeeper in Kafka
• Broker Cluster and Zoo...
Topics, partitions, and offsets in Kafka
• Topics, partitions, an...
Kafka Cluster with Multiple Brokers
• Kafka Cluster with Mul...
Topic with Replication in Multiple Broker Kafka Cluster
• Topic with Replication...
Setting up a single node Kafka Cluster on EC2
• Setting up a single n...
Kafka Log Segments in-depth intuition
• Kafka Log Segments in-...
Kafka Producer Key & Message Acknowledgements
• Kafka Producer Key & M...
Kafka Producer Internals
• Kafka Producer Internals
Kafka Topic Partitions & Producers using Python
• Kafka Topic Partitions...
Code:
-----------
from time import sleep
from json import dumps
from kafka import KafkaProducer
topic_name='hello_world'
producer = KafkaProducer(bootstrap_servers=['localhost:9092'],value_serializer=lambda x: dumps(x).encode('utf-8'))
for e in range(1000):
data = {'number' : e}
print(data)
producer.send(topic_name, value=data)
producer.flush()
producer.close()
Check this playlist for more Data Engineering related videos:
• Demystifying Data Engi...
Snowflake Complete Course from scratch with End-to-End Project with in-depth explanation--
doc.clickup.co...
🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY KZbin CHANNEL

[@robertgosz4281]

Thanks. That was actually good!

[@KnowledgeAmplifier1]

Glad to hear that Robert Gosz!!

[@naikmshweta]

I am getting BufferExhaustedException, we are sending 60k request, what is best way to solve this

[@user-or5gv9jt2i]

sir, i would want to do a little correction in code , that producer.flush() should come inside for loop, other wise its of no use. Thankyou !!!

[@AyushMandloi]

Thanks Man. From where u have learned all this ?

[@KnowledgeAmplifier1]

Hello Ayush Mandloi, I have learnt the basic concepts from Pluralsight courses & company's internal trainings , & eventually learnt the intermediate concepts while working on a real-project using Kafka & other Big Data Services for one of India's Largest Private Sector Bank ..

[@namangarg8654]

How can we check if the buffer is getting full or not?

[@olayoridickson]

Can you disable batching ? Is it required? Also Does is the buffer memory is on the client or server side?

[@user-or5gv9jt2i]

ITS ON THE CLIENT SIDE , BECAUSE IF LOADING TO SERVER FAILS THEN IT DOES A RETRY (THAT ALSO CAN BE CUSTOMIZED)IF EVEN AFTER TRYING FOR THOSE MANY TIMES IT FAILS , THEN ERROR IS POPPED ELSE ITS WRITTEN TO KAFKA CLUSTER