Scaling HANGFIRE: Processing More Jobs Concurrently

[@CodeOpinion]

Are you using Hangfire? Let me know in the comments what specifically for. If you have any questions about how to implement something with Hangfire, Let me know in the comments and I'll try and tackle it in another video.

[@maxkatz4254]

Thanks for the video! I have used Hangfire for delayed push notifications and for synchronizing two databases (NoSQL read-only and SQL for for write-read).

[@allannielsen4752]

Runs the reporting engine.

[@alexprift2242]

As always a good video with great content. Thanx Derek!

[@CodeOpinion]

Much appreciated!

[@gabrieliws]

Nice video, Derek ! I have been learning Hangfire and Quartz for the past few days. Imo HF is much easier to understand than Quartz. I have some questions, appreciate if you could give some info :) 1. What’s you thought on Quartz? 2. I also tried the HF dashboard, it is good but I wonder how to find a certain job based on recurring job id, since no filter in the UI. I feel it will be very difficult if I have massive number of jobs in production. 3. I plan to use multiple containers for HF servers and a single container for dashboard. I could think of hundreds/thousands of containers for the servers with multiple queues, do you think HF would be able to handle it? 4. I noticed HF persist the method name and arguments, so if we change the method structure it will be a breaking change and the existing jobs won’t be able to be executed. Do you have any tips on this matter? Apologise for the long questions. Thank you in advance 😀

[@CodeOpinion]

1-I haven't used quartz 2-not that I'm aware of. Maybe there is an extension? 3-Depends if your using different job storage, that's per dashboard.if each container has its own job storage then yes, dashboard per. 4-yes! If you change a signature of a job, it will break calling the job. You need to be backwards compatible for the existing job to run.

[@rajkumarks4443]

We're using Hangfire background scheduler in our .NET 6 Web API. As we're planning to host this app across multiple servers for load balancing, and these servers share a single database, are there any possibilities of job failures or dupes or running multiple times? Are there specific configurations like different schemas or queues that could help prevent such issues?

[@CodeOpinion]

A job will only run on one worker at a time. It handles concurrency for you.

[@malikbrahimi7504]

How does this compare to ResourceStack jobs on AKS?

[@sarathbaiju6040]

Hi, Is helper method available in hangfire library to get all the running job instances?

[@JoseVargas-dx7wz]

Hi, from the documentation I understood exactly what you showed in the video as ways to scaling the amount of jobs processed, but as soon as I'm setting the number of workers to more than 1, I experience that enqueued jobs are ran more than once. As if more than one thread is taking the same record from the list of enqueued jobs. I use HF 1.7.3 and mySQL local storage. Any toughts?

[@joymonscode]

Did you find any answer to this? I am considering HF for queuing jobs that must be processed only once.

[@asifshahzad7547]

Hi @CodeOpinion can you please share about how we can achieve DI in Hangfire for complex data types

[@asifshahzad7547]

@codeOpinion

[@CodeOpinion]

Hangfire Docs: docs.hangfire.io/en/latest/background-methods/passing-dependencies.html

[@shreyasmahajan2157]

Thanks a lot for the helpful video. I just wanted to ask a thing : I am planning to send around 2000 emails at once. What would be more beneficial : Worker Threads or Multiple Hangfire Servers ? Thanks for your help.

[@CodeOpinion]

Combination of both possibly. Ultimately how many threads you have per hangfire service is based on how much work (memory/cpu) is involved in processing the job/email.

[@shreyasmahajan2157]

@@CodeOpinion Thanks a lot. Will try with Worker Threads first.

[@reminvestor]

Can you explain the configuration difference between the application that is enqueuing the jobs vs the application that is running the hangfire server? From what I can tell you just don't start the server on the one that is enqueuing.

[@CodeOpinion]

If you're using MS DI, there is extension methods to configure server or just the client.

[@nqaiser]

If you happen to have multiple servers that can take work(e.g send an email to someone), how does hangfire ensure that no more than one server would execute the same job?

[@CodeOpinion]

Locks. Depends on the job storage but regardless locking.

[@fevziozgul6780]

Whatever you do with the hangfire scaling, if you have too many hangfire jobs running that put load on the main database, you will have bottlenecks and hangfire scaling will not help. What can one do in such situations?

[@CodeOpinion]

Yes, any downstream services, including database can be affected by scaling out more hangfire servers. In order to avoid a never ending backlog in the queue, you have to either have to scale all the downstream services affected which means process more messages or slow the rate down of messages are produced into the queue. Check out this video: kzbin.info/www/bejne/eHqqmn-Af9F4Zpo

[@thedacian123]

I think,i ve spotted here concurent consumer patther wit hangfire source acting as a message broker.Is this correct?

[@CodeOpinion]

Yes, competing consumers.

[@anitin3]

Hi Derek in your opinion which SPA Frontend framework is more popular in working with .net core apps ReactJS, Angular or Vue?

[@CodeOpinion]

I'm not entirely sure what's the most popular one. I posted a poll on Twitter: twitter.com/codeopinion/status/1316820854731427841

[@CodeOpinion]

This poll got a decent amount of votes. Remember this is what people are using, not what they necessarily want to use. Or at least that's how I intended it as being worded in the poll.

[@anitin3]

@@CodeOpinion thanks Derek.It was helpful i will start learning react.

[@sureshhalade]

Hi, Nice video. I am exploring this option and don't find any params that AddHangfireServer accepts. Is it tagged to any specific version of Hangfire ? Additionally, can we add additional worker threads and/or servers at runtime? And you did not show any code to add additional servers, can it be configurable? Thanks.

[@sureshhalade]

Look like we can do something like this: var options = new BackgroundJobServerOptions { WorkerCount = Environment.ProcessorCount * 5 }; app.UseHangfireServer(options); However how do we do all this at runtime to scale out and scale in back ? Any inputs? thanks.

[@CodeOpinion]

To increase the WorkerCount at runtime after it has already been started?

[@sureshhfreelance3917]

@@CodeOpinion yes

[@CodeOpinion]

@@sureshhfreelance3917 I don't believe you can after startup. At least from what I'm aware of. I'd have to look into Hangfire docs more to see if its possible.

[@sureshhfreelance3917]

@@CodeOpinion ok no worries. My understanding is also same. Thanks for response.

[@JeffChentingwei628]

Is polling the storage inefficient ?

[@CodeOpinion]

It's not ideal that's for sure. In my use cases, I want to process jobs quickly as possible, so the latency of the polling time is what concerns me with polling. The Hangfire.Redis.Pro (commercial) solves this by using Redis Pub/Sub so it's not polling and latency is really really low.

[@crisu85]

Nice video :) I'm also using HF to separate image processing from the main application (and scale by the number of servers). I'm wondering how you would approach to monitor HF servers? For example, my HF server just got killed by some exception or any other error (whole machine down). Would you do that by setting a watchdog on a different server or maybe any other clever solution? :)

[@CodeOpinion]

I currently use a combination of ASP.NET Core Health Checks and using the Hangfire Monitoring API to get the number of active servers, enqueued jobs, etc. Then I alarm based on those metrics.