Web-Queue-Worker Architecture Style for Scaling

[@cdnkarasu]

This exactly sums up our main product. I've always felt it doesn't quite fit monolith entirely and isn't microservices. Great overview of this valid style.

[@_rcs]

Thanks for always being so pragmatic and realistic.

[@CodeOpinion]

I try

[@GigAHerZ64]

I worked with a service exactly like that. We had sometimes huge spikes of incoming HTTP requests and we had to be able to take them in. At the same time, it was not particularly necessary to process them immediately. Our HTTP API endpoint was async with callback URL given in the request payload. Every request was independent and didn't depend on anything from other requests. In the HTTP request receiving part we created and saved a "processing context object" into CosmosDB and put the document ID into queue. (Azure Storage Queue) The workers listened on the queue and when any of them got an ID, it loaded the context object from the CosmosDB to start processing it. The worker itself had to call other external services and it allowed us to not flood our other partners as we had fixed amount of workers set up. The HTTP request receivers auto-scaled to hundreds, when needed. Queue scaled really well with small payloads, CosmosDB scaled really well with queries based on only primary keys, etc. Everything worked really well. We called it a "multi-instance monolith" or "multi-tier monolith".

[@CodeOpinion]

Nice! Queue workers can be helpful rate limiting as you mentioned for downstream.. assuming you limit how many messages processing concurrently. Beauty of a queue.

[@GigAHerZ64]

@@CodeOpinion correct! Forgot to mention that workers themselves limited their consumption rate. Good point!

[@juststudying1019]

Amazing channel

[@ErichHafenmaier]

Great video. I started using this pattern with the system I'm currently building/maintaining. We actually started using Hangfire in our .NET API for our background jobs (both scheduled and fire/forget) but quickly got into a situation where it became very difficult to work on locally due to several scheduled background jobs firing off once you ran the project locally. Decided to migrate those jobs to .NET worker services with Azure Service Bus to handle the messages and it's been a game changer. We typically use emails to do client notification but I'd like to try and incorporate some type of real-time streaming log of what the worker is doing but haven't got there yet.

[@piotrkowalski3460]

I do the same thing with Hangfire. I configured scheduled jobs to use Cron.Never when running on development environment

[@creamyhorror]

Honestly this just seems like the background queue/worker part of typical web frameworks. Rails, Laravel, etc. all have specific components for offloading slower tasks to workers via a work queue. There's also the notification step to handle, where the workers inform trigger a notification to the application or end user that the work is done so that follow-up steps can be started.

[@ArinSinabian]

I am not familiar with rails or a laravel. But I suspect the queue is not persisted in those frameworks if they don't use a database. The difference here is you use a broker which persist the messages and handles acknowledge from the worker. So if you restart your web server or worker the queue is still there and if the worker is half done you can continue processing. The toools provided by laravel or rails are of course useful but you must be aware if they have drawbacks. It might be useful for some cases and not for other.

[@pdgiddie]

At least for Rails, these jobs are persisted in the db. I think my main issue with these job queue features for web frameworks is that task code is quite fat/boilerplatey. Doing this with a message broker is more flexible in my experience. But ultimately the pattern _is_ pretty much the same.

[@andreashe36]

This video is like a dump from my mind. Great. So kinda acknowledgement that I am on the right way.

[@CodeOpinion]

Glad it could be of some validation!

[@Omer23007]

Just as usual great video! Thanks again for sharing

[@Tony-dp1rl]

I think every second video now on this channel is about queues

[@CodeOpinion]

There are a bunch that's for sure. Probably a combination of EDA, ES, Queues. I almost wanted to say I have more about boundaries. Curious, what would you like to see covered instead.

[@ernest1520]

This is very much how typically event-driven architecture is implemented, with one difference in that rather than using queues (for commands) they use topics/exchanges (for events). The overall flow is more or less the same.

[@CodeOpinion]

Yup, ultimately. Just to add to that you won't generally take an http API call and immediately publish an event without anything actually occurring. Meaning events are published based on something occurring. *You can model requests as events in some situations but that's another video. Thanks for the comment

[@tmbarral664]

pain point: yep, one. something you only slightly talked about : the job done feedback. Dunno about AWS, but on GCP, with a LB to spread the load, the WS will timeout after 30s. So if you plan to use WS to get notify, you're better have jobs running below 30s. Pretty annoying. if you have ideas how to circumvent this, I'd glad to hear it ;) Cheers.

[@williamkwesikwesh]

another way is to poll for it until its done

[@bernhardkrickl5197]

We use this architecture with a message bus to send back events from the worker to the clients.

[@CodeOpinion]

Yup! Another option are webhooks depending on if they accept them. Depends what your clients are.

[@thedacian123]

I did not understand how worker comunicates with the httpapi ,about the processing result.Thank you!

[@CodeOpinion]

Libraries used for websockets communication will abstract this from you if you're in a distributed environment where it uses an underlying backplane. For example with SignalR it will use Redis for this. So if you dispatch a message to a client/group from a worker, it's ultimately hitting the HTTP API that holds the websockets connection to the client.

[@thomaseichinger1717]

uuhh.. yeah! I like this pattern but it is very difficult.

[@RoyiKlein]

I am now going throgh a process of taking our Monolith and making it more robust and scalable, came across this cool video. Do you think this architecture can also work for Sync rest API? meaning splitting the application to Service side (API) and Workers where the API puts the request in a message queue and waits for a worker to publish a result and then returning this to the client?

[@CodeOpinion]

Yes. The biggest channel however is if you need that async work to notify the client that made the original request. You can often use websockets or some type of callback/webhook. Check out this video where I talk about it a bit: kzbin.info/www/bejne/aKq4n4xsZt-Xd80

[@sultanmohammad2135]

So, this entire system can be called a distributed system?

[@zumanoka3310]

Derek, is it possible to see the code for the patterns you discussed somewhere?

[@CodeOpinion]

You can access to that code if you join the channel. Not a lot of videos have code shown because it's generally more abstract and you can apply it depending on the language/platform you're using.

[@H0b0Gamers]

Would you have multiple queues for different types of jobs? For example, we are implementing a queue for generating large pdfs and its FIFO. Those pdfs are not needed immediately. I want to move other heavg logic to the queue, but that data is needed much faster. I can have thousands of pdfs generating and dont want to bottleneck the work i need done faster.

[@jandudulski5083]

I recommend to have queues based on SLA/SLO and scale each queue based on the requirements

[@andrewiecisa2907]

I use this pattern with the help of masstransit library. The only gotcha I found is the size of the payload may by greater than the message capacity.

[@BetrayedEnemy]

Im having the Same Problem. I Just Put the Data to be used in another place and referencing it inside of the Message

[@CodeOpinion]

Yup. Solution is the Claim Check Pattern: kzbin.info/www/bejne/ppawg5SeqZl2rcU