Doordash moves their Backend to Apache Kafka from RabbitMQ, VERY interesting! Let us discuss it!

Doordash the food delivery service has built an asynchronous task processing backend with Celery and RabbitMQ. They are having lots of outages and problems. Let us discuss how they solved their problem by moving to Apache Kafka.
Very well written article.
Resource
doordash.engin...
www.rabbitmq.c...
This work is not associated with or endorsed by VMware.
🏭 Backend Engineering Videos
• Backend Engineering (B...
💾 Database Engineering Videos
• Database Engineering
🛰 Network Engineering Videos
• Network Engineering
🏰 Load Balancing and Proxies Videos
• Proxies
🐘 Postgres Videos
• PostgresSQL
🚢Docker
• Docker
🧮 Programming Pattern Videos
• Programming Patterns
🛡 Web Security Videos
• Web Security
🦠 HTTP Videos
• HTTP
🐍 Python Videos
• Python by Example
🔆 Javascript Videos
• Javascript by Example
👾Discord Server / discord
Become a Member
/ @hnasr
Support me on PayPal
bit.ly/33ENps4
Become a Patreon
/ hnasr
Stay Awesome,
Hussein

[@oshosanyamichael9589]

CTO walks into the office tomorrow CTO: We are moving from rabbitmq to kafka Engineer: why is the move necessary? CTO: Doordash says rabbitmq is bad. Let's start moving.

[@hassanharera]

That is not correct, DoorDash doesn't say that, they said they have some requirements that RabbitMQ doesn't fit for

[@akay64x2]

@@hassanharera I think the guys' being sarcastic....

[@vinny142]

I like how pretty much every article in the style of "moving from X to Y" initially sounds like "Y is betterh than X, but later it turns out that the system designers either never thought that theywould grow to this size, or they never knew that X had a limitation, or some other random thing happened that makes Y more suitable than X, for *their* specific setup. It's interesting to learn why they wanted to move and hw they moved, but as you said at the beginning these articles do not tell you that one is better than the other, only that the other was more suitable for that particular application..

[@EddyCaffrey]

I love the videos that discuss others companies decisions. Thank you. Keep it up 👍🏾. My point of view Let’s not forget that these companies work with A LOT of asynchronous tasks. If they have decided to move from one technology to another, it just means that the current technologies do not meet their criteria anymore. For common life engineers and developers, these current technologies can work very fine. RabbitMQ is simple and easy for simple task like sending emails, sending notifications, image processing, and others.

[@abdulmobeen2477]

Agreed.

[@ragingpahadi]

Yes realtime problems help more than inverting binary tree on leetcode :p

[@abhishek-agarwal]

Great video Hussein. Your videos are so informative! Could you please make a video on Event Sourcing and CQRS?

[@section9999]

These case studies are awesome. It really helps to bridge that transition between knowing the concepts and being able o apply them successfully in production. More informacion please!

[@hnasr]

I totally agree! You can see the concepts in real production

[@jjames7206]

Thanks Hussein!! This is cool webside . Such Like :"Scaling Splunk Securely by Building a Custom Terraform Provider"

[@HarshBhikadia]

Minor correction - Celery does not open&close connection for each task request submission. The high conn. churn rate could be because high rate of server (producers) churn - when they have surge (auto-scaling kicking in) or something - or some sort of process-recycling feature (which would recycle the conn. as well) with server (uWSGI has this feature) which could be often and (worst if) at the same time (as this as often not randomised). This is guess-work by me - with whatever info available.

[@andreytamelo1183]

Super video! I applauded for $2.00 👏

[@hnasr]

❤️❤️

[@moritz5102]

25:40 Depending on the number of partitions and the number of consumers, a consumer can be assigned multiple partitions or even zero partitions (if I am not mistaken, 2 partition 3 consumers for example). So maybe „exactly one“ is unfortunate use of wording here. Nevertheless a partition is always assigned to exactly one consumer. EDIT: In context of a consumer group.

[@arlandmv4053]

Very good spanish my friend 😂

[@zainabedin1021]

Great video. Thanks for sharing knowledge.

[@blakejohnson4004]

Building it yourself slows you down, and Celery is a very strong task package. It allows you to get up and running quickly. And then developing tests for your celery tasks is much cleaner than building out a pub/sub package. I'm currently dealing with this nightmare. Celery allows you to bypass the message broker and call your methods directly. So you can run tests using pytest without running a rabbit machine. Some say that doesn't really matter, but to me, having to connect to a message broker for your tests to run is a pain in the ass at times. Also, flower is a UI that is built for celery so you can run your tasks and see what is happening on the backend. Again, not a lot of data there, but theres enough to know your system is running properly. If you NEED to build it yourself, then do so. But if you need speed, python probably isn't your move anyway

[@sariksiddiqui6059]

My question is the churnout problem they were facing with celery and how they resolved that with kafka is having multiple exectuors in parallel, behind kafka consumer, why couldnt do the same with celery?Why custom executors are working better than celery, whats the point of onboarding celery then?

[@hnasr]

Siddiqui Sarik my understanding is that celery doesn’t support kafka

[@sariksiddiqui6059]

@@hnasr thanks for the reply, my question is basically with celery. Given that rabbitmq + celery is a popular combo ,how come they have scaling issue?Is that because of the no of topics or something very specific to their business that celery just cant hanlde

[@devlifewithharsh7805]

I don't think closing a queue is a solution. You don't want the queue to be opened forever as that queue cannot process tasks related to another user

[@vaishnavsm]

Quick question: how do you come by all these articles? :thinking

[@hnasr]

Vaishnav Sreekanth Menon It is hard to find to be honest.. but I tailored google news so it serve me stuff I like.. and I report on the ones I like.. some news come from twitter

[@techwithameer]

Sometimes I can't even understand his videos. May be because I'm a fresher to this job

[@unkowndata2338]

Thank you for the video! Do you know what tool they used to make those architecture diagrams?

[@hnasr]

I know they are pretty good not sure what they use. Probably Google Draw? I use gliffy for my diagrams when I do any

[@phuctran6182]

Asynchronous doesn’t necessarily mean things are queued, that’s only true if asynchronous is implemented using event-loop. In a multi-threading environment, the async task will be separated to a different thread away from the main thread, and both of them will be processing at the same time.

[@sudharajamanickam7534]

Sorry for the basic question. active connections is always confusing to me..How many tcp connetions a client can have at a time with server(

[@hnasr]

Sudha Rajamanickam Not basic at all question is valid The 6-10 connection is just a browser arbitrarily limit to enable parallel request sending You can open as many connections as you want to a single host as long as 1) the client has enough memory 2) the server has enough memory 3) the client doesn’t exceed 2^16 source ports 4) the server doesn’t exceed 2^16 dest port This number is per source ip and destination ip So the server can open up 2^16 for every client But the server doesn’t usually have a single client it has many... the total number or connections for the server for all clients is limited to memory and cpu usage. Whatsapp single server handles 3 million tcp connections for example WhatsApp handles 3 MILLION TCP Connections Per Server! How do they do it? Let us discuss kzbin.info/www/bejne/rIKYoGethduIjsk

[@hackwithharsha5228]

Thank you Hussein !! Celery has configuration called broker_pool_limit, If set to None or 0 the connection pool will be disabled and connections will be established and closed for every use. Check docs for configuration option docs.celeryproject.org/en/stable/userguide/configuration.html#broker-pool-limit

[@hnasr]

Good find!! Thanks for sharing.. so you would need to set this connection pool to an appropriate number so it preheats and reuse connections.. I wonder if doordash team disabled this for a particular reason

[@hackwithharsha5228]

Hussein Nasser I hope, they might have disabled in the early stage of the project like any another mis-configuration settings or some other reason too.

[@MuztabaHasanat]

How does the request and response work while using message queue? Using REST it is easy. Suppose ServiceA called via REST ServiceB. And after processing the request ServiceB response with the process result. But when communicating between ServiceA and ServiceB using message queue how does it work? Because there is no response.

[@hnasr]

Muztaba Hasanat i do discuss the same question here Publish-Subscribe Architecture (Explained by Example) kzbin.info/www/bejne/hWKzmKSLpr9-iqM

[@pranavambhore9215]

You can store the task status in db. And have a GET api that will provide the status of that task.

[@zakstev786]

Excellent bro... Please keep up the good work

[@vinothshepard7451]

Meow

[@section9999]

agreed

[@matthewprestifilippo7673]

this guy literally didn’t know anything about this stack. why do a video in it?

[@killDJuice]

First 😂😂😂😂

[@slsoftshow]

I came to be first but you broken my heart. 🥺🥺🥺 😆😂😂😂

[@vinny142]

What is the point of that 'first' stuff?

[@TheDesvendador]

@@vinny142 they're needy

[@killDJuice]

@@vinny142 dk

[@tyrrelldavis9919]

@@vinny142 viewbots