Scalable & Event Driven Food Ordering App with Kafka & Python | System Design

In this video, I design a highly scalable backend system for a food ordering app. The system is event driven. Every important thing is an event, such as an order being placed and an order being confirmed.
These events are writtent to Apache Kafka which is a central bus for all moving data. Other systems, such as the transaction system or the email system, will be built on top of Apache Kafka. They will subscribe to different Kafka topics that they are concerned with, and process in real time as these messages are written to Kafka.
We use a library called kafka-python to hook up a locally running Kafka broker and our Python code.
The system should be highly scalable. Given the decoupled nature, you can scale up or down individual components as required. These individual components can either be microservices or some stream processing jobs or just plain Python files like I do here.
The system is also very extensible because of the nature of Apache Kafka. The messages for a given topic can be retained within Kafka. That means, if you were to build a new component, you could replay all the messages of a topic from the beginning and make the component process all messages from the beginning of the topic.
Similarly, given the persistence of Kafka, if there is an issue with any downstream or upstream system, you won't suffer from any data loss.
Other helpful videos:
[Apache Kafka Crash Course]( • Apache Kafka Crash Cou... )
[Microservice Communications]( • System Design - Micros... )
Further Resources
[Kafka Python package](kafka-python.readthedocs.io/e...)
[Kafka Docker Image](hub.docker.com/r/confluentinc...)
Code for the project: [Github Code](github.com/irtiza07/python-ka...)
#kafka #systemDesign #python
Visit me at: irtizahafiz.com
Or email me at: [e-mail]

[@imdadood5705]

Wow. This is a solid introduction for a person who has no idea what Kafka is. Thanks for this.

[@irtizahafiz]

Thank you! Glad you found it valuable.

[@chanduiit42]

Brilliant.. I am new to back-end engineering and most people just talk it..while you demonstrate it..awesome work..👏

[@irtizahafiz]

Thank you! Glad you got some value out of it. Let me know what else you think will be helpful for folks who are new to backend engineering.

[@DuyTran-ss4lu]

This is the coolest demostration. Thanks

[@irtizahafiz]

Glad you found it helpful : )

[@bjugdbjk]

Fantastic buddy !! KZbin took so long to recommend me this channel !!

[@irtizahafiz]

Eventually it did at least haha! Glad you found these helpful.

[@amanchandra2136]

One of the best demonstrative video for kafka using python. Thank you so much.

[@irtizahafiz]

Really glad to hear that! Will have more of these coming up : )

[@os365]

Definitely

[@voidisperfect]

thank you, amazing explanation! always struggled with understanding how kafka would work in real scenarios (used to overcomplicate it)

[@irtizahafiz]

Glad this helped! I have a few more Kafka videos in the work. Expect them to come out soon.

[@shamsularefinsajib7778]

Fantastic little explanation, I cannt be more grateful for the explanation and easy demonstration

[@irtizahafiz]

Glad you found it helpful! Thank you for watching my videos.

[@uchechukwumadu9625]

I needed this!

[@anirudhsyal5695]

Excellent video buddy great job

[@PavanKumar-pomskipper]

In one video , you have given fundamental working of kafka .. thank you

[@irtizahafiz]

Glad to hear that!

[@sohailhosseini2266]

Thanks for the video!!

[@Rileysnooker]

Great video, thanks! very clear

[@irtizahafiz]

Glad you found it helpful : )

[@shayankabasi160]

Awesome tutorial ...

[@quoccuongtran9185]

thank you so much

[@adityanjsg99]

Bundle of necessary things!

[@irtizahafiz]

Sorry you didn’t find it too helpful.

[@abessesmahi4888]

Clear as crystal Thank you so much for efforts Please be aware that music during tutorial are disturbing.

[@irtizahafiz]

Thank you so much for the feedback! Really appreciate it. Alright Imma lower the music volume significantly more for the next video. I would still like to keep it in the background to make the awkward silences better haha

[@abessesmahi4888]

@@irtizahafiz Your audience don't need any music, they need your knowledge. Look at the professionals like LinkedIn learning, Packtpub, O'reilly...etc, they don't put music in their training courses, this is on purpose. Anyway your are free to do whatever you want, it's your channel.

[@irtizahafiz]

@@abessesmahi4888 Alright you have convinced me haha! I will try the next few videos without music and see what the feedback is. Hopefully it's better for everyone : )

[@bharatgoyal9579]

amazing video, thanks

[@irtizahafiz]

Glad you liked it!

[@ashwanikapur]

Great explanation. Thanks for uploading it.

[@irtizahafiz]

Glad it was helpful!

[@ashwanikapur]

@@irtizahafiz Please remove distracting background music from your videos.

[@aniketwattamwar1514]

V Good video and explanation!!

[@irtizahafiz]

Glad you liked it!

[@NoobehPvP]

Great vid bro

[@irtizahafiz]

Appreciate it

[@shihabullah2475]

Very clear and awesome demonstration! Thank you for clearing my concept more as I have just started learning it. Can you please make a step by step kafka data pipeline setup tutorial using docker and any cloud infrastructure? You are doing great and should not stop helping people like us!

[@irtizahafiz]

I am glad you got value out of it. Don't forget to subscribe if you found it helpful :) Yes, I do have that video planned. I would expect that video to be out before the next month. No promises though.

[@shihabullah2475]

@@irtizahafiz that will be great. Eagerly waiting because still I didn't find any one in KZbin has made a step by step real-life implementation tutorial of making data pipeline using Kafka connect or whatever along with docker and azure or AWS. You will set the standard my friend. It will help so many upcoming data engineers and we will be grateful. Fingers crossed and praying so that you can manage time for it. 🤞❤️

[@irtizahafiz]

Haha that's good to know. Yes, I already started working on that a few days ago. Expect to see it soon :) Glad you are getting value out of my videos. That's the whole purpose!

[@shihabullah2475]

@@irtizahafiz eagerly waiting for the requested video! 🙏

[@mukeshnayal624]

Thank you brother for the beautiful explanation ! After watching this content my expectation going to increase in many fold. May I expect a project with fast api where celery, asynchronous, zookeeper, kafka or rabbitmq, micro Services will be use along with the deep explanation of each one.

[@irtizahafiz]

Hi! Thank you for watching. I did get a similar request before. I will try to come up with a video displaying all that soon.

[@mukeshnayal624]

@@irtizahafizExcited!

[@aurumlabs8271]

This was a great tutorial! I was wondering how I would scale a microservice like this! Would be a great tutorial.

[@irtizahafiz]

Glad it was helpful! Ideally, you would have multiple instances of each backend service running.

[@aurumlabs8271]

@@irtizahafiz That's what I was thinking! Thank you.

[@zendr0]

This is awesome. Can we have more of these? Please

[@irtizahafiz]

Haha! Yes yes I will try my best. Thank you for watching!

[@sribastavrajguru304]

Simple yet elegant. Amazing tutorial. One question I have here is, if one of the consumers(let's say email) was down for few minutes, will it loose the details it received during that time??Ideally it should not, right?? If yes, how do we achieve that?

[@irtizahafiz]

It won't lose anything. Kafka Consumers can store their own offset (current position). That means, when they go down, and then come back up, they will resume consumption from where they left off.

[@cybernetinc3775]

This tutorial is awesome. Thanks for it! I just have one question, In production, I don't think we would be running a while True loop. is there any other way to configure the listener? maybe using something like signals etc..

[@irtizahafiz]

Yes, there are multiple ways you can kick off Kafka Consumers.

[@pemmanalakshmi8513]

Hello Irtiza, This is the cleanest and best kafka hands-on video I have seen! 🙂 I have an use case required to be done, can you please suggest how to do it. We have an Flask application running on Kubernetes, I need to write a KafkaConsumer which reads the messages from a specific topic ID and perform some operation with the message fetched. Can we make the consuming part asynchronous(Background) so that it won't block the Flask app. Please feel free to suggest alternatives as well

[@irtizahafiz]

Hi! Glad you found some value out of it. Your Kafka Consumer should be its own standalone "thing" deployed on a different instance. Flask is a synchronous backend application, so it will be blocking if you deploy both in the same machine (unless you have multiple cores I think). Even then, ideally you want to deploy your Kafka Consumer independent of the Flask application.

[@nice-je3mp]

Thank you. If there's a lot of traffic on the real-time exchange, What do you think about writing kafka + Celery?

[@irtizahafiz]

Yup, that would work too. I think I talked about that in a different video.

[@manojphaju]

Hello Good explanation. Can you please do video on Docker and Kafka?

[@irtizahafiz]

Yes, soon

[@kspv2806]

thanks for the awesome content when we run the consumer file after the producer few orders are missed why is it so it should consume the previously generated ones as well , could you help me with this ?

[@irtizahafiz]

Sure. You can configure your Consumer to read from earliest message in the topic always. But you will want to make sure that re-processing the same message is idempotent.

[@wriiko]

in the github repository you have a consumer.py its not being used. Why? are u using the order_backend.py instead? Btw, excellent explanation! thanks

[@irtizahafiz]

Hi! Yes, I believe consumer.py has for demo purposes only. I am mostly using backend.py. Glad you found it helpful : )

[@retenim28]

hi, thanks for the video. It's the first approach to Kafka for me. Question: i want to add some functionality to this project. What can I do? Add some database or what? I'm a complete beginner

[@irtizahafiz]

Yup, you can always write to a DB from the Kafka topic.

[@retenim28]

@@irtizahafiz I man, i have a question. I tried to expand your project by taking data from an API created with Flask, that data is streamed into a kafka topic and then ingested to a Postgres DB. And in my project this works. But my question is: why do I need kafka in this case? I can simply read the API and write data directly in the DB. Sorry I am a little bit confused

[@ShinobuAe_]

bro can u make an video about building analytics systems like GA ..PLS..

[@irtizahafiz]

I have a project in works to build analytic systems using Apache Pinot and AWS Redshift.

[@mohammedadelhassan1198]

Is CDC good for the real scenario as orders should be stored in a database not pushed directly to Kafka ?

[@irtizahafiz]

Yeah, you can use CDC stream from the "order" table to trigger any post order actions.

[@rafaelg8238]

it's was great video. congrats. how can i change the 'ack' to '-1 or all' in this your example? in what files to do change? i don't wanna to lost any message.

[@irtizahafiz]

There's a way to do that when configuring your Kafka broker.

[@futhedude4848]

from 29:36 and so on, i don't quite understand the purpose of the video, did you just present the scalable of the kafa, which is the ability that kafka already have?

[@futhedude4848]

before 29:36 is just a normal kafka demo.

[@irtizahafiz]

Hi! Yes just a demo to show you how you can set up Kafka for domains like food ordering. In a production setting with Kafka deployed, it will be easy to scale up such an architecture.

[@uchennodim1161]

@code with Irtiza, thank you very much for making this video. Please I'll love to ask about the request Shihab Ullah made months ago. Any update on it? My company is currently migrating to a micro-service architecture and I have been charged with managing communication between the services with Kafka. All applications are deployed on AWS. So I would really love to watch your video on Apache Kafka with Python all hosted on AWS or other cloud technologies. Thank you so much. I really appreciate.

[@irtizahafiz]

Thanks for the idea! I can put something together in the coming months.

[@spartanarmy711]

Awesome tutorial, you have pushed 20k records and some are lost, so how to avoid that?

[@irtizahafiz]

You can configure Kafka to not drop any messages by setting the delivery semantic to "at-least-once"

[@blpraveen1436]

Can the order be received by kafka from client directly?

[@irtizahafiz]

Usually you go through a backend server. I am not aware of Kafka clients existing on the client.

[@karimporcari602]

You can technically do it, but the adress of you kafka brocker would be stored on the client side, and it's a really (really reeeaaaly) bad idea

[@sarveshwarsinghal5916]

32:47 transactions is the only one sending data in order_confirmed topic. email and analytics are consuming from order_confirmed topic. Transaction says last order id 19738 received, that means at max it will send 19738 order data to order_confirmed topic. But email and analytics show 19988. How is it possible, if the data is not send in order_confimed topic, how email and analytics are consuming it?

[@futhedude4848]

i think this is because he scroll the transactions terminal to the top instead of let it scroll to the bottom. so it just show the result at 19738 instead of 19988.

[@irtizahafiz]

That is correct! Thank you for helping.

[@RoHaN-vp6mh]

getting kafka timeout error , in the first 10 minutes of the video . wasted my 1 hour smh

[@irtizahafiz]

Sorry about that! Based on your system configuration, it might need some workaround.

[@sanganabasava7375]

hi very nice video, but i am getting error when i run any of either producer or consumer or order_backend AS raise Errors.NoBrokersAvailable() kafka.errors.NoBrokersAvailable: NoBrokersAvailable

[@irtizahafiz]

Thank you! Are you sure you have the Kafka broker running? If you are running it in a Docker container, make sure the ports exposed correctly so that your Producers and Consumers can actually listen to the topic.