Learning How to Consume Data from Kafka

Subscribe to show your support! goo.gl/1Ty1Q2 .
Patreon 👉🏽 / marceldempers
Today we take a deeper dive in Kafka and learn how to consume message. More importantly, how does message consumption work in Kafka ? We learn about Partitions and Partition keys.
Finally we also build our own consumer in Go and use a feature of Kafka called consumer group.
Checkout the source code below 👇🏽 and follow along 🤓
Also if you want to support the channel further, become a member 😎
marceldempers....
Checkout "That DevOps Community" too
marceldempers....
Source Code 🧐
--------------------------------------------------------------
github.com/mar...
If you are new to Kubernetes, check out my getting started playlist on Kubernetes below :)
Guide to Kafka for Beginners:
---------------------------------------------------
• Guide to Apache Kafka ...
Kubernetes Guide for Beginners:
---------------------------------------------------
• Kubernetes development...
Like and Subscribe for more :)
Follow me on socials!
marceldempers.dev
Twitter | / marceldempers
GitHub | github.com/mar...
Facebook | thatdevop...
LinkedIn | / marceldempers
Instagram | / thatdevopsguy
Music:
Track: jimmysquare - My Band | is licensed under a Creative Commons Attribution licence (creativecommon...)
Listen: / my-band
Track: Lumosik - [FREE USE] Chill, Sad type beat (prod. Lumosik) | is licensed under a Creative Commons Attribution licence (creativecommon...)
Listen: / free-use-chill-sad-typ...
Track: Omnibeats (trap beats & rap instrumentals) - Migos type trap beat - Sushi (trap beat with bells) | is licensed under a Creative Commons Attribution licence (creativecommon...)
Listen: / migos-type-trap-beat-s...
Track: Reckoner - lofi hip hop chill beats for study~game~sleep | is licensed under a Creative Commons Attribution licence (creativecommon...)
Listen: / reckoner-lofi-hip-hop-...
Track: calm boy - NIGHT TRAVEL | is licensed under a Creative Commons Attribution licence (creativecommon...)
Listen: / night-travel
Track: souKo - souKo - Parallel | is licensed under a Creative Commons Attribution licence (creativecommon...)
Listen: / parallel

[@nicknick65]

WOW! Absolutely brilliant. Thanks Marcel for a wonderful and thorough explanation. Learnt heaps. The format of your explanations with previewing the code, terminal split and somewhat doing it "on the fly" is so beneficial to witness the workflow. Love your work. You deserve many many more subs.

[@weitanglau162]

Any plan to make a video on Kafka on Kubernetes? :) Would also love a video on Kafka Connect and Kafka Stream!! Great explanation btw!

[@loganrobert5811]

i would also love this! I've been tooling around with strimzi but I'm such a kafka noob, it has been tough going.

[@tanweermakhani3348]

Amazing content. It's great to see how you build things and explain while building.

[@stefanschmidbauer6075]

Glad you brought back the good beats :)

[@adlisamy9900]

Hi Marcel, i love the kafka series, and i was wondering if you were planning to cover Kafka Connect as well as Schema Registry for the future videos? thank you again for the amazing job

[@jeromealtariba7339]

hi great content as usual, but wondering how to ensure resiliance and high availability if, by using the key, all messages are going to the same partition? What happens if this partation crashes ? How get the messages back and the last offset when rebuilding it? It sounds like having a cluster of 3 partitions is useless in this situation.

[@MarcelDempers]

Remember using a key per record only sticks that records messages to the same partition. Every record will have its own key so records are distributed. During a partition crash you will only lose records assigned to that partition, not all. BUT, this is where replication factor comes in. In the demo I set it to 1, for resilience you'll bump that number up to ensure there is replication so you dont lose the entire partition. It can become quite complex with a lot of trade offs to think about 💪🏽

[@jeromealtariba7339]

@@MarcelDempers thanks for your reply. yep I was thinking of the replication factor to fix this issue, but wasn't sure. looking forward to find a good tutorial on how implementing kafka in kubernetes. Searching around and it seemq not that trivial

[@felixl.7322]

Hey Marcel, your videos are great! I watched around the half of them on one day. I'm not joking, haha. Can you maybe make some videos about Zookeeper and about apache pulsar too? Would be nice. Thanks!

[@davidef.982]

Nice tutorial 👍

[@basavarajhiremath4627]

Great video! What happens when we have 2 brokers and 3 partitions? Do replicas mean the brokers? All these services are copies of the same Kafka container image, right? Also, how can we use a python script as a producer to generate the messages and the consumer to do some processing? Thanks in advance!

[@basavarajhiremath4627]

Also, you mentioned the messages are read from the beginning but in random order. I didn't quite catch this. Could you please help me understand this?

[@jaym7178]

Hi What are the tools I need to learn for a person hoping to be a data engineer I will start with python and sql any recommendations for tools I need to learn

[@georgelza]

Can I ask, know it's pretty simple, I'd say 90% of people doing this will be going in the direction of Confluent, can be wrong... how about doing this demo, but first standing up a confluent stack using their docker-compose build and then interfacing that, it's just slightly close to a most probable production template.