Bull & NestJS = Achieving Scale in Node.js

[@mguay]

Get my highly-rated Udemy courses at a discount here: michaelguay.dev/udemy/

[@Ballistic_Bytes]

Brilliant teaching quality and the amount of knowledge you have on the topics you teach is phenomenal.

[@mguay]

Thank you!

[@kerodfresenbetgebremedhin1881]

such quality content, I am familiar with most English speaking youtubers in this space and u r simply unmatched.

[@jorgecanas3668]

Totally agree with you

[@alvarorc28]

Hi Michael, very nice content!! Just want to point that you need to create a LB (eg nginx) to randomly pick a pod inside the k8s and make sure that you are indeed using a distributed system.

[@jrs_devs]

Amazing tutorial, new skill added to NestJS. I've done same but using PM2 instead of Kubernetes/Docker, for sake of simplicity.

[@innocentmazando3808]

For a newbie in Nest I didnt get quite a number but somehow this ia looking so cool and helpful.| Cant wait to flow in Nest and deployments with K8s like you did on this project. Thanks mate :)

[@brenohenrique2208]

This is a really incredible video. Thanks, bro.

[@mystic_monk55]

Thanks for the lessons 🙏

[@prashlovessamosa]

Thank you for making this.

[@hrachhovakimyan5112]

Unique content, keep rocking!!

[@buzzzom7384]

One remark here. Bull is not exactly the same thing as BullMQ. BullMQ is a newer version of Bull written in TS. NestJS provides separate packages for Bull and BullMQ.

[@josedesousa5789]

exactly !

[@ahmadmoussa7072]

100% 👍

[@NgocNguyen-zn6eo]

exactly, and the syntax is not completely the same.

[@janithrandeniya4504]

Subscribed. Great content, keep it up!

[@norhanel-nezamy6465]

great content keep rocking🚀

[@TheHardWorkingDollar]

Great guide. Like and subscribed.

[@brianjmquio5156]

Would love to see how are you going to write end-to-end test for this

[@nishus_vlog]

Great content. Thanks

[@muhammadhaseebkhalid2966]

Hi Micheal, great video!! Is it possible to run NestJS Bull Queues in a separate process on AWS/Heroku? If so, please provide an example.

[@m.keshavarzi3379]

hello, thanks for your excellent content and explanation, i have a question, if i write a processor and then i write a cron job to call this in a specific time, is it going to provide me the same functionality? thanks

[@laurentlouis3509]

Very useful, thank you!

[@user-tt6nc6mo7k]

Hi Michael, great video again. Very well put together, straight to the point with a real example. If you don't mind me asking, what are the reasons someone would choose to use bullmq over say rabbitmq? Is there a particular reason you have started looking into it?

[@mguay]

So I think they're different solutions to this same problem of achieving scale by distributed async processing. They obviously have their own differences between each other and it's a matter of finding what works best for you. BullMQ is "simpler" in my opinion, no need to ack messages, etc.

[@mr_ehmed]

i want to attached bull-board to monitor queues how i can do? i don't find any helping material

[@puntero]

This is great! I see however usage of Bull and Redis is mostly recommended for intensive tasks. Would implementing this queue architecture on, say a REST service, beneficial? I'm thinking maybe the queue system can help on handling errors and probably restarting failed tasks that run daily on the application, such as a data validation task or a database mutation.

[@mguay]

Agreed.

[@inanesilence]

Very nice content, but I would like to ask you. How to return data from the queue to one front end via websocket?

[@sportyarea6951]

Can we use kafka or rabbitmq instead of bullmq? Is that the right way

[@metatronicx]

Great tutorial. Thank you...But how do you set up the kubernetes cluster you're using ? Can you help me with that ?

[@himalayaahuja7547]

So instead of using Docker and K8s orchestration what if we're using a LB with multiple ec2 instances (like in AWS) would the entire setup still pick only 1 of the consumer to process the message and not multiple consumers (from other servers) fighting to process that message? I mean is this the nature of distributed queuing system that allows this to happen or something else?

[@ofuochi]

It's a distributed queue. Doesn't matter who or what consumes it. It is guaranteed (to a very high certainty) that it's gonna be sequential (FIFO).

[@kaiwenxiao7728]

Hi Michael, could you explain what difference between bull and rabiitmq? I’m an newer for backend, Thanks!

[@blender_wiki]

How you think this great solution compare to using an orchestrator like Netflix conductor or Netflix maestro? Thanks for sharing your knowledge with us.

[@thomassohet3619]

Great content again. Quick question though, with your first example "transcode an audio file", why choose to go for a job with BullMQ (the queueing system with Redis) instead of an event with EventEmitter (appart from showcasing it of course)? Both would achieve the same result right ? Not blocking the thread and decoupling the producer/emitter from the consumer/listener ? My question is thus : are those 2 patterns just different ways to implement a distributed system ? Why go for one or the other then ? What are the main differences ?

[@ofuochi]

There's is a fundamental difference between a QUEUE and a PUB/SUB or EVENTS. A queue (just like the name implies) strictly follows a FIFO sequence, meaning that, regardless of the number of replicas or instances there are, only one of this instance can dequeue and process the data in sequence (based on how they got enqueued). However, in the case of events (or pubsub), there's a producer of the event and potentially multiple subscribers (one to many). Hence, when an event is published, multiple instances could subscribe and react to that event. In essence, it depends on your architecture and what you're trying to achieve but that is basically the difference.

[@youjean83]

Just one thing to mention, nestjs/bull !== nestjs/bullmq!

[@technoinfoworldwide2329]

Can you amazon clone or any big projects using node microservice architecture and mongodb as a db.

[@mguay]

My new Ultimate Nest.js Microservices course will cover this! Stay tuned for its release in May.

[@moylababa8196]

Your tutorial is very good and advanced, if possible make a udemy course with all these topic

[@mguay]

My new Ultimate Nest.js Microservices course will cover this! Stay tuned for its release in May.

[@nilanjanmajumder9831]

why not use kafka ?

[@rakshiths.n9680]

When will you be launching your Udemy course, plz give us any update on that.

[@mguay]

I have about 3 hours recorded so far.

[@ekopurnomo9221]

waiting for full course from fundamental to advance

[@ConAim]

...need to set TTL on those bull:transcode..

[@patrik_stodola]

How to scale big PostgreSQL database with + 100 milion records?

[@Mnigos]

You are using bull not bullmq

[@GabrielGasp]

I believe the consumers should be an app separate from the API, no? The way your are doing in the video, if you have 5 replicas and send 5 requests, a 6th request will hang because the transcoding processes are blocking all 5 API nodes.

[@mguay]

Definitely - nice add. If we want to produce as fast as possible and not be affected by the consumers, this is a great architecture you describe.

[@raressh2712]

Well, actually, you can use the same app but offload the transcoding to a worker thread which is designed for CPU-intensive tasks, or you can spawn a process if you are using ffmpeg, for example. In this way you can put in the bull job the code that spawns the new thread/process and you will able to scale for an infinite number of concurrent transcodings, because bull will send the next job to process only after it finishes the previous one (unless you overwrite the default concurrency).

[@md.redwanhossain6288]

you are using bull and the video title says bullmq. This is not good.

[@mguay]

Thanks for pointing that out. I've updated it

[@franciscomartos2261]

This is not BullMQ

[@md.redwanhossain6288]

bullmq should be used instead of bull

[@mingli9563]

I don't like the nest.js. over engineered and complicated.