In-depth intuition on Topic Log Compaction in Apache Kafka

In this video let's explore Kafka Log Cleanup Policies in-depth.
Prerequisite:
-------------------
Kafka Log Segments in-depth intuition
• Kafka Log Segments in-...
Setting up a single node Kafka Cluster on EC2
• Setting up a single n...
Code:
------------
wget dlcdn.apache.org/kafka/3.4.0/...
tar -xvf kafka_2.13-3.4.0.tgz
To install Java --
----------------------------------------
java -version
sudo yum install java-1.8.0-openjdk
java -version
cd kafka_2.13-3.4.0
Start Zoo-keeper:
-------------------------------
bin/zookeeper-server-start.sh config/zookeeper.properties
Start Kafka-server:
----------------------------------------
Duplicate the session & enter in a new console --
export KAFKA_HEAP_OPTS="-Xmx256M -Xms128M"
cd kafka_2.13-3.4.0
bin/kafka-server-start.sh config/server.properties
It is pointing to private server , change server.properties so that it can run in public IP
To do this , you can follow any of the 2 approaches shared belwo --
1)Do a vi config/server.properties in insert mode -- change ADVERTISED_LISTENERS to public ip of the EC2 instance
2)You can modify the file using Winscp also
To create Log Compacted Topic:
--------------------------------------
cd kafka_2.13-3.4.0
bin/kafka-topics.sh --create --topic demo_testing3 --bootstrap-server 54.90.61.129:9092 --replication-factor 1 --partitions 1 --config cleanup.policy=compact --config min.cleanable.dirty.ratio=0.001 --config segment.ms=5000
To Start the Producer:
-------------------------
bin/kafka-console-producer.sh --topic demo_testing3 --bootstrap-server 54.90.61.129:9092 --property parse.key=true --property key.separator=,
To Start the Consumer:
-------------------------
cd kafka_2.13-3.4.0
bin/kafka-console-consumer.sh --topic demo_testing3 --from-beginning --bootstrap-server 54.90.61.129:9092 --property print.key=true --property key.separator=,
Check this playlist for more Data Engineering related videos:
• Demystifying Data Engi...
Apache Kafka form scratch
• Apache Kafka for Pytho...
Snowflake Complete Course from scratch with End-to-End Project with in-depth explanation--
doc.clickup.com/37466271/d/h/...
🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY KZbin CHANNEL

[@rohitkumar-p3y]

Wow! very clear explanation.

[@KnowledgeAmplifier1]

Glad you liked it @@rohitkumar-p3y, Happy Learning !

[@akshayaadinarayan533]

very clear explaination

[@KnowledgeAmplifier1]

Glad to hear this akshaya adinarayan! Happy Learning

[@AmarjeetAnandsingh]

This is amazing @KnowledgeAmplifier1 , such a clear and detailed explanation. I'm curious why on earth this video has sooo less views.

[@KnowledgeAmplifier1]

Thank you for your kind words, I am glad to hear the video is helpful to you!!

[@yoyoyoo100]

Hi Bro.. thanks for sharing your knowledge. I have been facing almost the same issue , its fixed by increasing the retention size. But now after running a long transaction commit ( > 1 lac records) my source debezium DB connector is unable to write any data to the kafka topic.. kafka connect logs show rebalance operation and going into hung state.. Please suggest how to handle this Issue.. Thanks in advance..

[@akshaybaura]

what is the impact of log compaction on consumer lag?

[@user-or5gv9jt2i]

SIR CAN YOU SHOW THIS FOR NORMAL PYCHARM, NOT EC2, IS IT POSSIBLE? OR JUS TXT THE CODE