Topics, partitions, and offsets in Kafka

Imp. points related to Kafka covered in this video :
-----------------------------------------------------------------------------------
1)Topics are particular stream of data in Kafka
2)You can have as many topics as you want
3)A topic is identified by it's name
4)Topics are split in partitions
5)Each partition is ordered
6)Each message within a partition gets an incremental id , called offset
7)Offset only have a meaning for a specific partition
8)Order is guaranteed within a partition (not across partitions)
9)Data is kept only for a limited time (default is one week)
10)Once the data is written to a partition , it can't be changed (immutability)
11)Data is assigned randomly to a partition unless a key is provided
Prerequisite:
--------------------
Install and run Apache Kafka & integration with Python using kafka-python
• Install and run Apache...
Multiple Producer & Multiple Consumer in a Kafka Topic
• Multiple Producer & Mu...
Intuition on Log files in kafka & Kafka Brokers
• Intuition on Log files...
Broker Cluster and Zookeeper in Kafka
• Broker Cluster and Zoo...
Code used in this video is available in the comment section.
Check this playlist for more Data Engineering related videos:
• Demystifying Data Engi...
Snowflake Complete Course from scratch with End-to-End Project with in-depth explanation--
doc.clickup.com/37466271/d/h/...
🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY KZbin CHANNEL

[@KnowledgeAmplifier1]

Codes used in the demo : -------------------------------------------- Start Zookeeper: --------------------------- F:/kafka_2.12-3.2.0/bin/windows/zookeeper-server-start.bat F:/kafka_2.12-3.2.0/config/zookeeper.properties Start Kafka-server: ----------------------------- F:/kafka_2.12-3.2.0/bin/windows/kafka-server-start.bat F:/kafka_2.12-3.2.0/config/server.properties Create a topic: ------------------------- F:/kafka_2.12-3.2.0/bin/windows/kafka-topics.bat --create --topic demo_testing --bootstrap-server localhost:9092 --replication-factor 1 --partitions 3 Start Producer: -------------------------- F:/kafka_2.12-3.2.0/bin/windows/kafka-console-producer.bat --topic demo_testing --bootstrap-server localhost:9092 Start Consumer: ---------------------------------- F:/kafka_2.12-3.2.0/bin/windows/kafka-console-consumer.bat --topic demo_testing --from-beginning --bootstrap-server localhost:9092 Reading message from specific partition: --------------------------------------------------------------------- F:/kafka_2.12-3.2.0/bin/windows/kafka-console-consumer.bat --topic demo_testing --from-beginning --bootstrap-server localhost:9092 --partition 2 Reading message from specific offset inside a specific partition: --------------------------------------------------------------------------------------------------------------- F:/kafka_2.12-3.2.0/bin/windows/kafka-console-consumer.bat --topic demo_testing --bootstrap-server localhost:9092 --partition 2 --offset 1 Reading message from specific offset : ------------------------------------------------------------------------- F:/kafka_2.12-3.2.0/bin/windows/kafka-console-consumer.bat --topic demo_testing --bootstrap-server localhost:9092 --offset 0

[@user-bf4kg9iw7y]

Explained well. concepts are so clear. Below are the topics learned so far - 1. Verifying multiple partitions in the server logs directory 2. Checking the offset based on the flag parameter within the partition 3. Content check for each message within the partition

[@bigdataprofessional1880]

Crisp explanation, 100% perfect

[@KnowledgeAmplifier1]

Thank you BigData Professional! Happy Learning

[@user-or5gv9jt2i]

absolute genius. amazing xplaination sir, thankyou so much.

[@KnowledgeAmplifier1]

Thanks and welcome

[@ayansrivastava731]

if someone is using kafka >2.4 , this approach of round robin(random partitioning) will NOT work, instead there is something called sticky partitioner where you'll see all msgs going to same partition. WHY? because we didn't specify keys in the start. The partitioning strategy by kafka has changed.

[@KnowledgeAmplifier1]

You are correct Ayan , here are the videos where I have explained the same -- kzbin.info/www/bejne/Y5bSZXyGitibiNEsi=o_tQF0TYGE5qFO_D

[@BasitAIi]

Thanks, I was also thinking about the same problem

[@user-vp4by6sb1m]

Thank you Sir

[@KnowledgeAmplifier1]

You are @user-vp4by6sb1m ! Happy Learning

[@shobhitarya1637]

Wow, you explained the topics very well with practical as well as with internals. Its a very good tutorial for learning kafka. Thank you .

[@KnowledgeAmplifier1]

Thank you for your kind words Shobhit Arya! Happy Learning

[@kamalnayan9157]

Great work man! Thank you

[@KnowledgeAmplifier1]

Thank you Kamal Nayan! Happy Learning

[@ohuuthai1678]

Why does my data only save to partition 0 when I was like you, I tried adding many other input data examples, but it only jumps to partition 0

[@digitaltechconnect1318]

Please provide us the session for Confluent Kafka

[@vigneshr4197]

One kafka producer,two consumer subscribes to kafka topic then how partition works?

[@KnowledgeAmplifier1]

Hello Vignesh R , you can have a look in this video , I explained your scenario in detail -- kzbin.info/www/bejne/r3zdlaijlNuLbc0 Hope this will be helpful! Happy Learning

[@alapysalapylalapy2352]

Can we connect?

[@KnowledgeAmplifier1]

Hello Namrata Pathak, please post your doubt here , If I know that topic or have any idea , I will surely reply asap , also for any kafka related doubts , please check this playlist once , might help you if the topic is covered in the playlist already -- kzbin.info/aero/PLjfRmoYoxpNrs0VmIq6mOTqXP52RfZdRf

[@alapysalapylalapy2352]

@@KnowledgeAmplifier1 hi i am facing one error with my assignment where to post the picture??

[@alapysalapylalapy2352]

@@KnowledgeAmplifier1 what are the rules of data ingestion i.e Kafka to hdfs in which video it is present??