This exactly sums up our main product. I've always felt it doesn't quite fit monolith entirely and isn't microservices. Great overview of this valid style.
@_rcs3 ай бұрын
Thanks for always being so pragmatic and realistic.
@CodeOpinion3 ай бұрын
I try
@GigAHerZ643 ай бұрын
I worked with a service exactly like that. We had sometimes huge spikes of incoming HTTP requests and we had to be able to take them in. At the same time, it was not particularly necessary to process them immediately. Our HTTP API endpoint was async with callback URL given in the request payload. Every request was independent and didn't depend on anything from other requests. In the HTTP request receiving part we created and saved a "processing context object" into CosmosDB and put the document ID into queue. (Azure Storage Queue) The workers listened on the queue and when any of them got an ID, it loaded the context object from the CosmosDB to start processing it. The worker itself had to call other external services and it allowed us to not flood our other partners as we had fixed amount of workers set up. The HTTP request receivers auto-scaled to hundreds, when needed. Queue scaled really well with small payloads, CosmosDB scaled really well with queries based on only primary keys, etc. Everything worked really well. We called it a "multi-instance monolith" or "multi-tier monolith".
@CodeOpinion3 ай бұрын
Nice! Queue workers can be helpful rate limiting as you mentioned for downstream.. assuming you limit how many messages processing concurrently. Beauty of a queue.
@GigAHerZ643 ай бұрын
@@CodeOpinion correct! Forgot to mention that workers themselves limited their consumption rate. Good point!
@juststudying10193 ай бұрын
Amazing channel
@ErichHafenmaier3 ай бұрын
Great video. I started using this pattern with the system I'm currently building/maintaining. We actually started using Hangfire in our .NET API for our background jobs (both scheduled and fire/forget) but quickly got into a situation where it became very difficult to work on locally due to several scheduled background jobs firing off once you ran the project locally. Decided to migrate those jobs to .NET worker services with Azure Service Bus to handle the messages and it's been a game changer. We typically use emails to do client notification but I'd like to try and incorporate some type of real-time streaming log of what the worker is doing but haven't got there yet.
@piotrkowalski34603 ай бұрын
I do the same thing with Hangfire. I configured scheduled jobs to use Cron.Never when running on development environment
@creamyhorror3 ай бұрын
Honestly this just seems like the background queue/worker part of typical web frameworks. Rails, Laravel, etc. all have specific components for offloading slower tasks to workers via a work queue. There's also the notification step to handle, where the workers inform trigger a notification to the application or end user that the work is done so that follow-up steps can be started.
@ArinSinabian3 ай бұрын
I am not familiar with rails or a laravel. But I suspect the queue is not persisted in those frameworks if they don't use a database. The difference here is you use a broker which persist the messages and handles acknowledge from the worker. So if you restart your web server or worker the queue is still there and if the worker is half done you can continue processing. The toools provided by laravel or rails are of course useful but you must be aware if they have drawbacks. It might be useful for some cases and not for other.
@pdgiddieАй бұрын
At least for Rails, these jobs are persisted in the db. I think my main issue with these job queue features for web frameworks is that task code is quite fat/boilerplatey. Doing this with a message broker is more flexible in my experience. But ultimately the pattern _is_ pretty much the same.
@andreashe363 ай бұрын
This video is like a dump from my mind. Great. So kinda acknowledgement that I am on the right way.
@CodeOpinion3 ай бұрын
Glad it could be of some validation!
@Omer230073 ай бұрын
Just as usual great video! Thanks again for sharing
@Tony-dp1rl3 ай бұрын
I think every second video now on this channel is about queues
@CodeOpinion3 ай бұрын
There are a bunch that's for sure. Probably a combination of EDA, ES, Queues. I almost wanted to say I have more about boundaries. Curious, what would you like to see covered instead.
@ernest15203 ай бұрын
This is very much how typically event-driven architecture is implemented, with one difference in that rather than using queues (for commands) they use topics/exchanges (for events). The overall flow is more or less the same.
@CodeOpinion3 ай бұрын
Yup, ultimately. Just to add to that you won't generally take an http API call and immediately publish an event without anything actually occurring. Meaning events are published based on something occurring. *You can model requests as events in some situations but that's another video. Thanks for the comment
@tmbarral6643 ай бұрын
pain point: yep, one. something you only slightly talked about : the job done feedback. Dunno about AWS, but on GCP, with a LB to spread the load, the WS will timeout after 30s. So if you plan to use WS to get notify, you're better have jobs running below 30s. Pretty annoying. if you have ideas how to circumvent this, I'd glad to hear it ;) Cheers.
@williamkwesikwesh3 ай бұрын
another way is to poll for it until its done
@bernhardkrickl51973 ай бұрын
We use this architecture with a message bus to send back events from the worker to the clients.
@CodeOpinion3 ай бұрын
Yup! Another option are webhooks depending on if they accept them. Depends what your clients are.
@thedacian1233 ай бұрын
I did not understand how worker comunicates with the httpapi ,about the processing result.Thank you!
@CodeOpinion3 ай бұрын
Libraries used for websockets communication will abstract this from you if you're in a distributed environment where it uses an underlying backplane. For example with SignalR it will use Redis for this. So if you dispatch a message to a client/group from a worker, it's ultimately hitting the HTTP API that holds the websockets connection to the client.
@thomaseichinger17173 ай бұрын
uuhh.. yeah! I like this pattern but it is very difficult.
@RoyiKlein19 күн бұрын
I am now going throgh a process of taking our Monolith and making it more robust and scalable, came across this cool video. Do you think this architecture can also work for Sync rest API? meaning splitting the application to Service side (API) and Workers where the API puts the request in a message queue and waits for a worker to publish a result and then returning this to the client?
@CodeOpinion13 күн бұрын
Yes. The biggest channel however is if you need that async work to notify the client that made the original request. You can often use websockets or some type of callback/webhook. Check out this video where I talk about it a bit: kzbin.info/www/bejne/aKq4n4xsZt-Xd80
@sultanmohammad21353 ай бұрын
So, this entire system can be called a distributed system?
@zumanoka33103 ай бұрын
Derek, is it possible to see the code for the patterns you discussed somewhere?
@CodeOpinion3 ай бұрын
You can access to that code if you join the channel. Not a lot of videos have code shown because it's generally more abstract and you can apply it depending on the language/platform you're using.
@H0b0Gamers3 ай бұрын
Would you have multiple queues for different types of jobs? For example, we are implementing a queue for generating large pdfs and its FIFO. Those pdfs are not needed immediately. I want to move other heavg logic to the queue, but that data is needed much faster. I can have thousands of pdfs generating and dont want to bottleneck the work i need done faster.
@jandudulski508323 сағат бұрын
I recommend to have queues based on SLA/SLO and scale each queue based on the requirements
@andrewiecisa29073 ай бұрын
I use this pattern with the help of masstransit library. The only gotcha I found is the size of the payload may by greater than the message capacity.
@BetrayedEnemy3 ай бұрын
Im having the Same Problem. I Just Put the Data to be used in another place and referencing it inside of the Message
@CodeOpinion3 ай бұрын
Yup. Solution is the Claim Check Pattern: kzbin.info/www/bejne/ppawg5SeqZl2rcU