Topics, partitions, and offsets in Kafka

Imp. points related to Kafka covered in this video :
-----------------------------------------------------------------------------------
1)Topics are particular stream of data in Kafka
2)You can have as many topics as you want
3)A topic is identified by it's name
4)Topics are split in partitions
5)Each partition is ordered
6)Each message within a partition gets an incremental id , called offset
7)Offset only have a meaning for a specific partition
8)Order is guaranteed within a partition (not across partitions)
9)Data is kept only for a limited time (default is one week)
10)Once the data is written to a partition , it can't be changed (immutability)
11)Data is assigned randomly to a partition unless a key is provided
Prerequisite:
--------------------
Install and run Apache Kafka & integration with Python using kafka-python
• Install and run Apache...
Multiple Producer & Multiple Consumer in a Kafka Topic
• Multiple Producer & Mu...
Intuition on Log files in kafka & Kafka Brokers
• Intuition on Log files...
Broker Cluster and Zookeeper in Kafka
• Broker Cluster and Zoo...
Code used in this video is available in the comment section.
Check this playlist for more Data Engineering related videos:
• Demystifying Data Engi...
Snowflake Complete Course from scratch with End-to-End Project with in-depth explanation--
doc.clickup.com/37466271/d/h/...
🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY KZbin CHANNEL

[@KnowledgeAmplifier1]

Codes used in the demo :

[@user-bf4kg9iw7y]

Explained well. concepts are so clear.

[@bigdataprofessional1880]

Crisp explanation, 100% perfect

[@user-or5gv9jt2i]

absolute genius. amazing xplaination sir, thankyou so much.

[@ayansrivastava731]

if someone is using kafka >2.4 , this approach of round robin(random partitioning) will NOT work, instead there is something called sticky partitioner where you'll see all msgs going to same partition. WHY? because we didn't specify keys in the start. The partitioning strategy by kafka has changed.

[@user-vp4by6sb1m]

Thank you Sir

[@shobhitarya1637]

Wow, you explained the topics very well with practical as well as with internals. Its a very good tutorial for learning kafka. Thank you .

[@kamalnayan9157]

Great work man! Thank you

[@ohuuthai1678]

Why does my data only save to partition 0 when I was like you, I tried adding many other input data examples, but it only jumps to partition 0

[@digitaltechconnect1318]

Please provide us the session for Confluent Kafka

[@vigneshr4197]

One kafka producer,two consumer subscribes to kafka topic then how partition works?

[@alapysalapylalapy2352]

Can we connect?