Daniel Earwicker Chief Software Architect
FISCAL Technologies Ltd

Hangfire - A Tale of Several Queues

C# HANGFIRE 2019-05-24

If you've used Hangfire you know it's a really quick and easy way to give your app a queue of durable background jobs, with automatic retrying and a very nifty dashboard to let you see what's happening right now. Jobs can trigger further jobs and so a complex series of processing stages can be decoupled and spill out into a queue of little units of work.

You can setup one database (such as Redis) to store the state of all your jobs, and then multiple identical workers can attach to that database and munch through the jobs, taking them through the lifecycle:

[Enqueued] -> [Processing] -> [Finished]

Or maybe:

[Enqueued] -> [Processing] -> [Failed] -> [Enqueued] -> [Processing] -> [Finished]

It's all good. Once enqueued, either the job finishes or you'll be able to see it in the dashboard and read all the lovely exceptions.

I recently hit the problem of having multiple tenants (totally separate customers in this case) executing jobs in the same Hangfire instance. This works fine until a low priority tenant floods the queue with jobs and stops higher priority work from getting done.

Under-the-Sea Analogy

You run a service that washes sea creatures. A whale stopped by because he wants the krill cleaned out of his mouth and also his tail could use a service. It's not that urgent though and you aren't charging him as he's a friend. You get started on that, but then a shark comes along (he happens to be one of your best customers) and he needs his teeth cleaned for a photo session starting in ten minutes. Oh no!

What you need are two different queues, like a fast lane and slow lane. I could have just said it's like a grocery shop with a "5 items or fewer" sign at the checkout, but I've already written all that stuff about the sea creatures and I'm not deleting it now.

Hangfire queues and their limitations

So how can Hangfire help us here? It has the concept of queues. When you start a "server" (nothing to do with a physical box or even a process: you can start multiple of them inside one process) you specify what queue(s) it will read from and how many dedicated worker threads it has. So you could start separate servers for fast and slow, each dedicated to their one queue.

But there's a problem. by design, Hangfire doesn't assign jobs to queues. Rather, when a job is enqueued, a queue name such as fast can (optionally) be specified. The choice of queue is not stamped on the job, but stored as a property inside the state object representing the Enqueued state. This means that when the job transitions to Processing it will be picked up by the fast server, but now nothing is storing the queue name. Why is that a problem?

Well, jobs fail sometimes, and they need to be retried. In fact this is part of what makes a system like Hangfire so valuable; occasional transient problems (a database glitch) don't stop your work from getting done. But obviously when that happens to our jobs, we want them to be enqueued for their second attempt on the same queue as last time.

Extension points

Hangfire supports several extension points, and the relevant ones here are:

By combining these we can basically brute-force jobs into running on the queue we've stamped them with.

So the IClientFilter might look like this:

public class QueueNameRecorder : IClientFilter
    public void OnCreating(CreatingContext filterContext)
        var enqueuedState = filterContext.InitialState as EnqueuedState;

        if (string.IsNullOrWhiteSpace(enqueuedState?.Queue))
            filterContext.SetJobParameter("stashedQueueName", enqueuedState.Queue);

    public void OnCreated(CreatedContext filterContext) { }

And in the start-up of your clients:

GlobalJobFilters.Filters.Add(new QueueNameRecorder());

This means that if we enqueue a job on the fast queue:

_backgroundJobs.Create<Toothbrush>(j => j.CleanTeeth(), new EnqueuedState("fast"));

the above OnCreating handler will kick in and store the value fast in a custom property stashedQueueName, permanently associating the job with that queue.

Meanwhile in the worker processes, you need the IElectStateFilter:

public class QueueNameFixer : IElectStateFilter
    public void OnStateElection(ElectStateContext context)
        var queueName = context.GetJobParameter<string>("stashedQueueName");

        if (context.CandidateState.Name == "Enqueued" &&
            context.CandidateState = new EnqueuedState(queueName);

... registered in GlobalJobFilters exactly the same way as before, but in worker processes. And OnStateElection will fire whenever any job is changing to any new state. Here we're only interested in jobs that are becoming Enqueued and that have a stashedQueueName.

What you then see in your Hangfire dashboard is a bit odd - the job will first be enqueued on the DEFAULT queue and then immediately enqueued a second time on the correct queue.

Now you're all set to clean a fish, or whatever it was I said before!

Hangfire - A Tale of Several Queues 2019-05-24
Factory Injection in C# 2019-05-16
How Does Auth work? 2018-11-24
From Ember to React, Part 2: Baby, Bathwater, Routing, etc. 2018-03-18
From Ember to React, Part 1: Why Not Ember? 2017-11-07
json-mobx - Like React, but for Data (Part 2) 2017-02-15
Redux in Pieces 2017-01-28
Box 'em! - Property references for TypeScript 2017-01-11
TypeScript - What's up with this? 2017-01-01
MobX - Like React, but for Data 2016-12-28
Eventless - XAML Flavoured 2016-12-24
Immuto - Epilogue 2016-12-20
Immuto - Radical Unification 2016-09-22
Immuto - Working with React (An Example) 2016-09-16
TypeScript - What is a class? 2016-09-11
Immuto - Strongly Typed Redux Composition 2016-09-11
TypeScript and runtime typing - EPISODE II 2016-09-10
TypeScript and runtime typing 2016-09-04
What's good about Redux 2016-07-24
TypeScript multicast functions 2016-03-13
Introducing doop 2016-03-08
TypeScript is not really a superset of JavaScript and that is a Good Thing 2015-07-11
A new kind of managed lvalue pointer 2014-04-27
Using pointer syntax as a shorthand for IEnumerable 2014-04-26
Adding crazily powerful operator overloading to C# 6 2014-04-23
Introducing Carota 2013-11-04
Want to comment on anything? Create an issue!