What is a Kafka Consumer and How does it work?

Now that your data is inside your Kafka cluster, how do you get it out? In this video, Dan Weston covers the basics of Kafka Consumers: what consumers are, how they get your data flowing, and best practices for configuring consumers in a real-time data streaming system. You will also learn about offsets, consumer groups, and partition assignment.
Learn more about Apache Kafka here with our free courses: developer.confluent.io/course...
RELATED RESOURCES
► What is a Producer and How does it work: • What are Kafka Produce...
► Kafka Consumers Explained: • Kafka Consumers Explai...
► Consumer Group IDs: • Introduction to Consum...
► Apache Kafka 101-Consumers: cnfl.io/49JMXum
► Apache Kafka for .NET Developers: cnfl.io/3wJWdzY
► Kafka Consumer | Documentation: cnfl.io/3VebfYT
CHAPTERS
00:00 - Intro
00:08 - What is a Consumer?
01:26 - Consumer Offsets
02:15 - Consumer Group
02:49 - Partition Assignment
03:46 - Consumer Configuration
04:55 - Closing
--
ABOUT CONFLUENT
Confluent is pioneering a fundamentally new category of data infrastructure focused on data in motion. Confluent’s cloud-native offering is the foundational platform for data in motion - designed to be the intelligent connective tissue enabling real-time data, from multiple sources, to constantly stream across the organization. With Confluent, organizations can meet the new business imperative of delivering rich, digital front-end customer experiences and transitioning to sophisticated, real-time, software-driven backend operations. To learn more, please visit www.confluent.io.
#streamprocessing #apachekafka #kafka #confluent

[@ConfluentDevXTeam]

This is Dan, just a minor correction to clarify what I say in the video. Both the key and value are serialized on the producer and deserialized once they arrive at the consumer. After the serialization happens then the key, if any, is used to determine the partition the messages will go to. However, this is only done after the message has been serialized. Similar to the producer video we just published, this video is meant to give an overview of Kafka Consumers. Be sure to check out our Kafka Producers video as well: kzbin.info/www/bejne/mXipm5dtpN1nnpY I’d also love to hear the other aspects of Kafka you’d like to see us breakdown, or areas you feel a lot of people struggle with.

[@SagarD-sg8ej]

In the 4 min of producer and 5 min of consumer given me good knowledge

[@USL-hd9cf]

Great video! Thank you for the clear explanation. Can you provide more insight on how the key is used to serialize and deserialize messages in Kafka? I would also like your thoughts on consumer group strategy. You said it could range from zero to the number of partitions. What is a good way to determine the number of consumer groups.

[@ConfluentDevXTeam]

Thanks for your comment. Kafka uses Murmur2 to generate the hash key. It then uses a mod function to determine the partition. Here is the function inside of Kafka: github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/BuiltInPartitioner.java#L327 You can also check out this blog post to dive deeper into how the hashing works: www.confluent.io/blog/standardized-hashing-across-java-and-non-java-producers/ As for consumer group strategy, this blog post by my colleague Lucia is a great primer to understand more about consumer groups: www.confluent.io/blog/kafka-consumer-group-partition-strategy/ Let me know if these resources help!

[@BishalKarki-pe8hs]

Need schema-registry with python tutorials

[@SagarD-sg8ej]

In the 4 min of producer and 5 min of consumer given me good knowledge