Setting up a single node Kafka Cluster on EC2

Get a single node Kafka cluster up and running in next to no time!
For steps , check the pinned comment.
Prerequisite:
--------------------
Install and run Apache Kafka & integration with Python using kafka-python
• Install and run Apache...
Multiple Producer & Multiple Consumer in a Kafka Topic
• Multiple Producer & Mu...
Intuition on Log files in kafka & Kafka Brokers
• Intuition on Log files...
Broker Cluster and Zookeeper in Kafka
• Broker Cluster and Zoo...
Topics, partitions, and offsets in Kafka
• Topics, partitions, an...
Kafka Cluster with Multiple Brokers
• Kafka Cluster with Mul...
Topic with Replication in Multiple Broker Kafka Cluster
• Topic with Replication...
Check this playlist for more Data Engineering related videos:
• Demystifying Data Engi...
Snowflake Complete Course from scratch with End-to-End Project with in-depth explanation--
doc.clickup.co...
🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY KZbin CHANNEL

[@KnowledgeAmplifier1]

Steps : ------------- Launch one EC2 Instance & install kafka-- wget dlcdn.apache.org/kafka/3.2.0/kafka_2.12-3.2.0.tgz tar -xvf kafka_2.12-3.2.0.tgz To install Java -- ----------------------- java -version sudo yum install java-1.8.0-openjdk java -version cd kafka_2.12-3.2.0 Start Zoo-keeper: ------------------------------- bin/zookeeper-server-start.sh config/zookeeper.properties Start Kafka-server: ---------------------------------------- Duplicate the session & enter in a new console -- export KAFKA_HEAP_OPTS="-Xmx256M -Xms128M" cd kafka_2.12-3.2.0 bin/kafka-server-start.sh config/server.properties It is pointing to private server , change server.properties so that it can run in public IP To do this , you can follow any of the 2 approaches shared belwo -- 1)Do a vi config/server.properties in insert mode -- change ADVERTISED_LISTENERS to public ip of the EC2 instance 2)You can modify the file using Winscp also Create the topic: ----------------------------- Duplicate the session & enter in a new console -- cd kafka_2.12-3.2.0 bin/kafka-topics.sh --create --topic demo_testing2 --bootstrap-server {Put the Public IP of your EC2 Instance:9092} --replication-factor 1 --partitions 1 Start Producer: -------------------------- bin/kafka-console-producer.sh --topic demo_testing2 --bootstrap-server {Put the Public IP of your EC2 Instance:9092} Start Consumer: ------------------------- Duplicate the session & enter in a new console -- cd kafka_2.12-3.2.0 bin/kafka-console-consumer.sh --topic demo_testing2 --bootstrap-server {Put the Public IP of your EC2 Instance:9092}

[@hedonicas]

This is one of the best videos about Kafka on KZbin. Thank you sir!

[@KnowledgeAmplifier1]

You are welcome Burak Kılıç! Happy Learning

[@moizdewaswala6717]

Thanks for explaining every small detail.

[@ye__in]

You are the best teacher! Thank you so much.

[@KnowledgeAmplifier1]

You are welcome 예부인 YEBUIN! Happy Learning

[@handesubodh]

Very useful video, after long frustration this video helped me,,,, THANK YOU FOR THE VIDEO

[@KnowledgeAmplifier1]

Glad to know the video is helpful to you Subodh Hande! Happy Learning

[@ShakeelAhmed-xo3xz]

Very good and well explained

[@KnowledgeAmplifier1]

Glad you liked it!

[@cloudandsqlwithpython]

this is really great work Sir

[@KnowledgeAmplifier1]

Thank You!

[@flesz_]

nice do you have videos on how to use it with certificates or TLS encryption , rather than plaintext ? Also Kafka connect would be nice to see and using mirror maker

[@danielarroyave3183]

Hi :) Can you explain us in another video how to set up kakfa and zookeeper as a service systemd? Thanks!

[@shreyas-s5g]

sir as of now i know only some basic linux commands, kindly publish a full playlist on linux from scratch if possible.

[@JimRald-v3f]

Hi brother, How we can install Kafka cluster among multiple servers or EC2 instances and run Odoo ERP ap on that cluster. Thanks.