Apache Kafka 101: Partitioning (2023)

► TRY THIS YOURSELF: cnfl.io/kafka-101-module-1
Learn how partitioning works in Apache Kafka. With partitioning, the effort behind storing, processing, and messaging can be split among many nodes in the cluster.
► For a COMPLETE IMMERSIVE HANDS-ON EXPERIENCE, go to cnfl.io/kafka-101-module-1
- - -
ABOUT CONFLUENT
Confluent, founded by the creators of Apache Kafka®, enables organizations to harness the business value of live data. The Confluent Platform manages the barrage of stream data and makes it available throughout an organization. It provides various industries, from retail, logistics, and manufacturing, to financial services and online social networking, a scalable, unified, real-time data pipeline that enables applications ranging from large-volume data integration to big data analysis with Hadoop to real-time stream processing. To learn more, please visit confluent.io
#kafka #kafkastreams #streamprocessing #apachekafka #confluent

[@vinod187118]

finally i found the partition logic in this video. Thanks a lot for the crisp video tim.

[@azharmobeen]

It's simply a great explanation. Thanks Man

[@sanjayb0886]

Very well done Tim - Thank you!

[@philipackerley5775]

Yet again Tim, rock solid short snappy overview.

[@tomydurazno6243]

Great video guys, very helpful!

[@yohannistelila8879]

Thank you, crystal clear!

[@unny2708]

Thanks Tim.. This is best video over internet for those who just jumped in to Kafka....

[@timmyho1269]

Thank so much, it is so clearly

[@varunraj3736]

Very well explained ..thank you

[@Neosam01]

man your style of explanation is just awesome..I mean how can you explain things so easily that too without much animation or something...one of the best instructor I must say

[@Sulerhy]

incredible visualized video. Thank you so much

[@vladvlad2798]

Amazing explanation!!

[@igorburilo3937]

good explanation, thanks!

[@nazirzhumakhan]

clear explanation, thanks

[@Anbu_Sampath]

Best explanation for partition.

[@aditya-desai]

What is the relation between events and messages? Messages have key-value pairs? or Events have key-value pairs? what exactly ... Events or messages are stored in partitions with key-value pairs? this was a helpful video Take key ------> HASH Function-------> Output mod (Total no. of partitions) ------> Resulting no. is Partitions number where message going to store. I was unaware of this concept. Thank You Tim.

[@matifibrahim]

Thanks to animation in this video now I better understand partition

[@RaushanKumar-co3wj]

awesome .

[@ireallylikeshoes]

Excellent explanation, but one thing that I see just about all videos lack is explaining WHY partitioning is useful and when it is not.

[@ConfluentDevXTeam]

Gilles here from Confluent. It's a good point, I was wondering the same thing last week and realised we don't have a video that goes into details of how you should partition your topics and why. The best resource is still this blog post from Jun Rao: www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster/. Hope this helps!

[@kurtvicious9375]

hey tim, can you tell me another advantage using partition

[@facusana]

what if we want to store information for each day in the NASDAQ stock market with ~3000 symbols and 1 billon of trades per day. Should we use one topic for each symbol or just one topic with handred of partitions? I want to understand a real case with a hugh amount of data!

[@yang120000]

When consumer read the topic, how does it know which partition to read the message out?

[@TymexComputing]

Hi - what do the Fire logo and wheel logo mean? Are these logos of old Kafka versions? I only could find a slideshow "Kafka - Past, present and future" :)

[@azharmobeen]

Is it possible one message can belongs to multiple partitions ?

[@alen740926]

so it is generally correct , if not always, to say that messages in different partitions within same topic are mutually exclusive?

[@nimasarayan4531]

What's the name of this song? ps: Kafka is amazing

[@8eck]

So if you want a FIFO queue, you are limited to only one partition?

[@glennbullock3778]

When I subscribe to a partitioned Topic, I still get all the messages eventually, just not necessarily in the correct order, right?

[@victorsterling6806]

Why don't use consistent hashing but use hash/ num_partition?

[@lianglu4442]

If the number of partitions changes, does Kafka re-hash and redistribute events? If not then events with the same ID could end up in different partitions?

[@zampogna]

It would be great to have a crisp explaination also for that

[@gerooq]

Why would events have the same ID

[@padmanathanramasamy8354]

Do we need to mentioned the number of partition while creating the topics? For example if I create the key with CustomerID and create the Kafka topic with 5 Partition then when there is a 6th Customer comes with different Key then which Partition will stored?

[@AmitMahto-bf5np]

Yes,you have to mention no. of partitions.

[@kamal-xd7id]

What if one partition node goes down, then the order can be messed up?

[@masoudhosseini8672]

in my humble opinion it is the case in real world, to have a customer acting as a noisy neighbour.

[@pavelkouzmitcheu7965]

Since Kafka 2.4 the Default Partitioner is set to Sticky Partitioner, and not Round Robin. Hence, if you wonder why all the messages with the null key land in the same partition this is the reason. If you want to set it to Round Robin to how it was before version 2.4 then set it in the props passed to ProducerFactory: props.put(ProducerConfig.PARTITIONER_CLASS_CONFIG, RoundRobinPartitioner.class);