Kafka Tutorial - Broker Configurations

Spark Programming and Azure Databricks ILT Master Class by Prashant Kumar Pandey - Fill out the google form for Course inquiry.
forms.gle/Nxk8...
-------------------------------------------------------------------
Data Engineering using is one of the highest-paid jobs of today.
It is going to remain in the top IT skills forever.
Are you in database development, data warehousing, ETL tools, data analysis, SQL, PL/QL development?
I have a well-crafted success path for you.
I will help you get prepared for the data engineer and solution architect role depending on your profile and experience.
We created a course that takes you deep into core data engineering technology and masters it.
If you are a working professional:
1. Aspiring to become a data engineer.
2. Change your career to data engineering.
3. Grow your data engineering career.
4. Get Databricks Spark Certification.
5. Crack the Spark Data Engineering interviews.
ScholarNest is offering a one-stop integrated Learning Path.
The course is open for registration.
The course delivers an example-driven approach and project-based learning.
You will be practicing the skills using MCQ, Coding Exercises, and Capstone Projects.
The course comes with the following integrated services.
1. Technical support and Doubt Clarification
2. Live Project Discussion
3. Resume Building
4. Interview Preparation
5. Mock Interviews
Course Duration: 6 Months
Course Prerequisite: Programming and SQL Knowledge
Target Audience: Working Professionals
Batch start: Registration Started
Fill out the below form for more details and course inquiries.
forms.gle/Nxk8...
--------------------------------------------------------------------------
Learn more at www.scholarnes...
Best place to learn Data engineering, Bigdata, Apache Spark, Databricks, Apache Kafka, Confluent Cloud, AWS Cloud Computing, Azure Cloud, Google Cloud - Self-paced, Instructor-led, Certification courses, and practice tests.
========================================================
SPARK COURSES
-----------------------------
www.scholarnes...
www.scholarnes...
www.scholarnes...
www.scholarnes...
www.scholarnes...
KAFKA COURSES
--------------------------------
www.scholarnes...
www.scholarnes...
www.scholarnes...
AWS CLOUD
------------------------
www.scholarnes...
www.scholarnes...
PYTHON
------------------
www.scholarnes...
========================================
We are also available on the Udemy Platform
Check out the below link for our Courses on Udemy
www.learningjo...
=======================================
You can also find us on Oreilly Learning
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
=========================================
Follow us on Social Media
/ scholarnest
/ scholarnesttechnologies
/ scholarnest
/ scholarnest
github.com/Sch...
github.com/lea...
========================================

[@ScholarNest]

Want to learn more Big Data Technology courses. You can get lifetime access to our courses on the Udemy platform. Visit the below link for Discounts and Coupon Code. www.learningjournal.guru/courses/

[@sohamkamanitech]

These tutorials are awesome! Straight to the point, no non-sense 🤘

[@santoshdash7646]

Best ever explanation on Kafka. Thank you very much Sir for this tutorial.

[@mayankarora5824]

If anybody wants to learn the Apache Kafka in a step-by-step approach then you are there on the right link. The way the concept has been explained in the videos is recommendable and worth watching. Thank you for uploading such tutorials. Looking for your videos on more topics.

[@nehagupta5100]

A beginners' go-to tutorial for Kafka.Explanation of the concepts in very simple words.

[@melwynjensen]

One of the best tutorial I have seen on KZbin. Hats off!

[@Belioyt]

Thank you for this video tutorial series. The concepts are well explained and there is a clear narrative from video to video that builds up on the topics.

[@BestowTechs]

I really wanna appreciate the way you teach straight to the point, very simple language, wonderful examples....... I have never see any tutorial like this before.. Thank you ...

[@ScholarNest]

My pleasure.

[@DeepakSharma_youtube]

I agree, you are a very good teacher.

[@pritpalsingh5676]

excellent, nicely explained. Looking fwd to new playlists (hadoop, linux, spark, cassandara, elastic search and many more)

[@simonwaffleman]

Deleting topics is also good for dynamic queue needs. Auto-create and delete in these instances is critical to have enabled.

[@ScholarNest]

agreed.

[@SPeri6]

so much content i browsed for Kafka yours is the best thanks a ton !

[@rakeshdeb91]

Can you show how to create topic by using topic management tool?

[@rameshthamizhselvan2458]

I'm following all your videos and I thought all your videos are enough for the Kafka developers to shine up with their career. but I still find some important things are missing in your videos like saving offsets. kindly request you to post the videos regarding preserving the offset in databases.

[@SubhashPavuskarhackzleo]

I have 2 query 1. What is the default size when producer pushes message buffer to the Kafka. 2. What is the parameter need to set in the properties in case needs to increase the size of message buffer.

[@lokeshsetty]

Wonderful tutorials. Thank you for helping me understand all kafka topics so easily. Can you please a few videos on managing security in Kafka?

[@supriyasawant8498]

Nicely explained..very useful..Thank you !

[@KC-zn4gt]

Clearly explained. Great tutorials and highly recommended.

[@sudhirshetty5724]

NICE, Pls also make tutorial on Cassandra and Elasticsearch

[@ScholarNest]

Sure, C* is one of my favorites. Let me complete Spark and Scala.

[@dawednesru5353]

Thank you so much Sir!

[@shivamgoel92]

Thanks for awesome videos, I have a question, will consumer and producer which we will create will connect to a specific partition, eg: do we need to specify in a producer which partition it needs to write to ? and same for consumer which partition it needs to listen to ?

[@ckudalkar]

Simply Awsome!!!! God Bless You!!!

[@prithivirajan1599]

Excellent and easy understand

[@mustakimpatvekar9781]

So when you say start cleanup on reaching of the criteria, does it clean all the messages in the partition or just the oldest until the criteria condition is false again

[@elfyjohnson261]

What happens if the zoo keeper machine goes down? Is there a backup machine for zoo-keeper too?

[@ScholarNest]

Yes. In a typical case, we have three zookeeper nodes.

[@happyworld6637]

Excellent!!! Very useful.

[@bunthaideng2492]

You should include the link you mentioned in the description.

[@ScholarNest]

Everything is. Available at www.learningjournal.guru

[@balakrishnag7478]

Can you please post tutorials for git, gradle, jenkins and hudson.

[@ramasubbareddy6147]

Question in this video is when many brokers are connected to zookeeper, can't be zookeeper overloaded to manage all the kakfa brokers right?

[@ScholarNest]

You can scale the zookeeper as well.

[@ibrahimelkilani8508]

If I have, say, 3 partitions with replication factor 3. Now what I understood is that they have all to connect to the same zookeeper. Ok what if they can't due to network issues ? Will the replication continue when the network is avaialble again?

[@ScholarNest]

If a replica fails to catch-up with the leader, it is removed from the ISR list. They can enter to the ISR once they again catch-up all the messages.

[@prakshaldoshi2521]

Hi Sir, I have one query. You mentioned that log.retention.ms/bytes configuration is used for clean up the message from the partition. What if the message is still not consumed by the consumer, will Kafka clean up that message?

[@ScholarNest]

We normally set retention period as 7 days. You can set it longer. Still not read yet, then loose it.

[@techenthusiast2920]

Hi sir , I am using ambari ,I try to push the msgs in console produce.sh After pushing the several msgs into producer it shows like failed to meta data update , but I'm using the port 6667 please help me sir

[@manishpandey3780]

Nice session👌👌👌

[@srinivasyadav9925]

Lets say i have 100 VMs where in brokers are running. On how many VM’s zookeeper need to run? If on only one vm where should be that?

[@raghavendraraghu5811]

how to find kafka brokers are up and running or not ? @learning journal please answer

[@princepaulson1]

sir how to upgrade brokers. how do we know the current versions

[@Mert-ed4xv]

everything is perfect, but auto German subtitle?

[@AVISH747]

Really good Sessions :)

[@csharma8732]

Thank you sir.

[@aritrabasu8794]

Thank you pm sir

[@hugodeiro]

Hi! I'm watching all your video tutorials about Kafka. They're very good. Congratulations! But I have a question. You said that we use Zookeeper to centralizes the cluster (each node knows the existence of the other by ZK), how we could manage it if Zookeeper's node goes down? Do we have any leader-follower treatment to treat this?

[@ScholarNest]

We normally setup zookeeper quorum of 3 or 5 nodes to manage that risk.

[@hugodeiro]

Do you explain in any of those videos how to do it?

[@josephkingstonleomariamich7220]

Super!

[@arunbm123]

Sir These Configuration should be made in all machines?

[@ScholarNest]

yes

[@srinivasyadav9925]

Does same topic need to be created on each VM?

[@ScholarNest]

No

[@cu7695]

Once you create the topic on any broker. It is available on all brokers if they are configured as listener in server.properties

[@kailasandhale8873]

You should put these courses on Udemy too.