Kafka Consumers in Python - A Walkthrough

[@JacobChencha]

I came to learn Kafka, learnt that and how to write great python code!!

[@fleshka48]

That is absolutely wonderful explanation! Thank you!

[@michaelslavev]

This is briliantly explained. As a kafka admin of multiple cluster, we are now bit scared of multiple teams trying to implement consumers and producers by themself because of poor offset management. I should make some Python/JS/Java examples! Thanks.

[@northjosh1]

This is incredible. Came to learn some Kafka but ended up learning great Python. Thanks a lot!

[@souravbarua3991]

Great video. Waiting for this and upcoming videos. Thank you for this tutorial series 🙏.

[@krisajenkins]

Glad you're enjoying it. We have quite a few planned, but if there's anything in particular you're interested in seeing, let me know. :-)

[@souravbarua3991]

@@krisajenkins For now I am learning from you. Eagerly waiting for next step/videos.Thank you again for this tutorial series🙏.

[@samable9585]

fantastic! will have to watch your other videos - thank again!

[@vincentchong6005]

OMG. I actually understand what you are talking about! Thank you sir.

[@QuixStreams]

You're welcome!

[@steverosam1265]

Another great video Kris

[@krisajenkins]

Thanks! 🙂

[@dailypakaffairs2590]

Really a great video. I was waiting for such masterpiece on kafka with python, can't find any before. Can you also make a video on how to add a notifier_consumer (may it through email or something). Also consider configuring Kong with kafka.

[@radoniainarakotonirina2856]

Nice vids 🎉

[@romankhmaladze5804]

Very nice, thank you!

[@munishkhatri33]

Which code editor are u using in this video ?

[@krisajenkins]

It’s Neovim. 👍

[@munishkhatri33]

@krisajenkins Thanks, I tried to install it, but it doesn't give type hints as I saw in your video. Any plugins required for that?

[@krisajenkins]

@@munishkhatri33 Ah yeah, takes a little configuration. `nvim-lspconfig` is what you need, but if I were you I'd start with the Neovim Kickstart project. It's a default config which will set up lsp for you, plus a lot more.

[@krisajenkins]

(I'd put a link to it, but every time I do, KZbin auto-deletes my comment. Google will guide you to it. 👍)

[@vipulsarode2722]

Great video, thanks! Can you tell me briefly how I can perform transformations in the consumer using Pandas, in real-time?

[@QuixStreams]

Glad you enjoyed it! There's a video here that goes into performing realtime stream transformations with a panda-like API: kzbin.info/www/bejne/a6TUlpqvlKqDl5I It's note exactly the same as Pandas, but it the underlying data's stored in Kafka it'll probably solve your problem in the same way. 🙂

[@DataSurfer]

I go through how to migrate Pandas code to Quix Streams code in a video. Hopefully it provides some inspiration: kzbin.info/www/bejne/fHSQdaWEl8ponKM

[@yifeitong]

Hello, thanks for the video. Really helpful to get start to understand Kafka. I was working in server and successfully created the topic learnt from previous video, there I managed to write non-duplicate messages. Following this video, when subscribing the topic, I got no messages (None), but I do see a message in that topic. i.e. my topic is 'weather_data_demo-Yifei', when subscribing, there is a line printed: [DEBUG] [quixstreams] : Assigning topic partition "weather_data_demo-Yifei[0]", but message is always None, any hints on this ? Many thanks

[@yifeitong]

I see, because the producer hasn't sent messages, with 'auto_offset_rest=latest' will only read new messages sent after it starts. 'earliest' will work, but after consuming the messages once, running script again, no messages obtained..

[@sridharstreakssri]

@@yifeitong same issue here. found any solution?

[@yifeitong]

@@sridharstreakssri try auto_offset_reset earliest ?

[@DePhpBug]

I'm still new to Kafka , if i use socket io , should this socket io server sit in Kafka? to manage it?

[@QuixStreams]

Apache Kafka is essentially a distributed server running in a cluster. You coudn't host your socket.io server inside of Kafka but it could run in the same infrastructure (one option is in Quix Cloud, which can run any Docker container). From there you could use a socket.io client in Quix Streams

[@DePhpBug]

@@QuixStreams Thanks :)

[@wagyumedia]

insta subscribed

[@michelchaghoury9629]

i have a question, will it gonna listen for ever if i am in a case were i want to have this code written to be always listenning, and never stops, if i run that code and leave it will it works as i want or i need to do something in addition? @Quix

[@QuixStreams]

Hi, this code will run forever, unless there is an issue like the machine it's running on being turned off, it's also not production code so adding some error handling etc would certainly be a good idea. If you want to run this in production with resiliency and SLA's check out Quix Cloud -> quix.io/signup