Kafka Cluster with Multiple Brokers

In this video, we are going to set up a Kafka Cluster in Local with 3 Kafka Brokers.
Prerequisite:
--------------------
Install and run Apache Kafka & integration with Python using kafka-python
• Install and run Apache...
Multiple Producer & Multiple Consumer in a Kafka Topic
• Multiple Producer & Mu...
Intuition on Log files in kafka & Kafka Brokers
• Intuition on Log files...
Broker Cluster and Zookeeper in Kafka
• Broker Cluster and Zoo...
Topics, partitions, and offsets in Kafka
• Topics, partitions, an...
Code used:
---------------------
Start Zookeeper:
------------------
F:/kafka_2.12-3.2.0/bin/windows/zookeeper-server-start.bat F:/kafka_2.12-3.2.0/config/zookeeper.properties
Start Kafka-server:
--------------------
F:/kafka_2.12-3.2.0/bin/windows/kafka-server-start.bat F:/kafka_2.12-3.2.0/config/server.properties
F:/kafka_2.12-3.2.0/bin/windows/zookeeper-shell.bat localhost:2181 ls /brokers/ids
F:/kafka_2.12-3.2.0/bin/windows/kafka-server-start.bat F:/kafka_2.12-3.2.0/config/server1.properties
F:/kafka_2.12-3.2.0/bin/windows/kafka-server-start.bat F:/kafka_2.12-3.2.0/config/server2.properties
F:/kafka_2.12-3.2.0/bin/windows/kafka-server-start.bat F:/kafka_2.12-3.2.0/config/server3.properties
Create a topic:
----------------
F:/kafka_2.12-3.2.0/bin/windows/kafka-topics.bat --create --topic demo_testing2 --bootstrap-server localhost:9092,localhost:9093,localhost:9094 --replication-factor 1 --partitions 5
Start Producer:
-----------------
F:/kafka_2.12-3.2.0/bin/windows/kafka-console-producer.bat --topic demo_testing2 --bootstrap-server localhost:9092,localhost:9093,localhost:9094
Start Consumer:
----------------
F:/kafka_2.12-3.2.0/bin/windows/kafka-console-consumer.bat --topic demo_testing2 --from-beginning --bootstrap-server localhost:9092,localhost:9093,localhost:9094
Sample Messages used in the demo:
------------------------------------------------------------------
Hi , this is the first message--second server , partition 0
Hello 2nd one--partition4 , first server
Hi there third one--partition3 , second server
4th message coming---partition 2, third server
5 th message testing--second server , partition 0
test1223--partition1 , first server
Welcome--partition4 , first server
Check this playlist for more Data Engineering related videos:
• Demystifying Data Engi...
Snowflake Complete Course from scratch with End-to-End Project with in-depth explanation--
doc.clickup.com/37466271/d/h/...
🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY KZbin CHANNEL

[@njt3773]

Very clear explanation. And you're not scared of showing errors on demo which shows your confidence in the field! Nice work

[@KnowledgeAmplifier1]

Thank you njt3! Happy Learning

[@user-bf4kg9iw7y]

Thank you for the detailed explanation. You have explained all the internal process in more detail. Loved the series :)

[@KnowledgeAmplifier1]

Glad it was helpful avinash basetty! Happy Learning

[@lasandun]

Great video. Explains the theory and shows it in practice.

[@KnowledgeAmplifier1]

Thank you Lahiru Lasandun! Happy Learning

[@ChandanaandSanthosh]

Very useful content! Thank you

[@KnowledgeAmplifier1]

Glad it was helpful!

[@PawanSharma-dd7jv]

Awesome sir. I have never seen such a descriptive videos you have made. Very very clear explanation. I loved it. 100 out of 100 from me. Do you also take online classes? Truly want to learn more big data stuff from you only. ❤

[@KnowledgeAmplifier1]

Hello Pawan Sharma, Thank you for your inspiring comment! As of now , I am not providing any private online classes , all my videos are available in this channel for free ... If you want to explore Kafka in depth using Python , you can refer this playlist -- kzbin.info/aero/PLjfRmoYoxpNrs0VmIq6mOTqXP52RfZdRf Happy Learning

[@bhushankorg5606]

Thanks your channel is so helpful. Can you show mirrormaker replication once.

[@truongvannhan954]

Great video , It will be even better if you can make a video for kraft mode

[@chenguo8767]

Thanks for the great lecture. If one server is down and the producer still keeps publishing messages, are these new messages going to be randomly sent to other partitions in healthy brokers, so I assume there will be no message loss, right? You also mentioned key which can be used to specify which partition the message is going to. what if there are keys associated with messages, but their corresponding partitions are down. what happen to these messages? is there an automatic adjustment mechanism in Kafka?

[@KnowledgeAmplifier1]

Hello Chen Guo , if one server is down and the producer still keeps publishing messages , the new messages will be going to same partition which is available in ISR List in another broker , basically to make the whole system fail-safe & reliable , one partition data is replicated in other brokers , for details , please refer this video -- kzbin.info/www/bejne/oJvXeICfn5J3htE&feature=shares Hope the video will make your doubts clear ... If you have further doubts after watching that please post in comment section , I will try to reply back asap .. Happy Learning

[@kunaldinkar3868]

Hello Sir, I have configured 3 servers with 4 partitions but all my producer messages are going in partition3 in server1. What is the reason that other partitions are empty with only one partition storing messages? Thankyou in Advance

[@AmeerDiaz]

same thing happening with me also. I think new version of Kafka does it with batch

[@BasitAIi]

Sticky partitioning

[@digitaltechconnect1318]

Post the code notes for the practice. we haven't seen here

[@KnowledgeAmplifier1]

Hello Digital tech connect, I forgot to post he codes for this video , I posted the codes(used in the video) in the description box.. Happy Learning

[@digitaltechconnect1318]

provide me the notes for execution

[@KnowledgeAmplifier1]

Hello Digital tech connect, the code is available in the description box.. Happy Learning

[@digitaltechconnect1318]

provide us this class notes for practice

[@KnowledgeAmplifier1]

Hello Digital tech connect, the code is available in the description box.. Happy Learning