[Feature Request] Job throttling

[profsmallpine]

This might already be possible, but I haven't been able to figure it out after reading the docs. If I only want a specific type of job to run 1 at a time for example, is this possible?

[mfrister]

It looks like unique jobs go in the direction of what you want and I'm looking for something similar: A way to ensure a job doing an expensive operation can't have multiple concurrently running instances, so it doesn't put too much load on another system.

UniqueOpts with ByState look like they can almost do this. Explicitly setting the default values ensures there's no job inserted as long as there's another one in the DB, including completed jobs. Removing JobStateCompleted from the list sounds like it would do what what we want, but the documentation says (highlights by me):

The list may be safely customized to add additional states (cancelled
or discarded), though only retryable may be safely removed from the
list.

Warning: Removing any states from the default list (other than retryable)
forces a fallback to a slower insertion path that takes an advisory lock
and performs a look up before insertion. This path is deprecated and should
be avoided if possible.

So I'm thinking maybe I shouldn't use this approach. Is there a better one?

[mastercactapus]

I’m looking for a good solution to this as well. Ideally, I’d like a ByState: [Pending] mechanism that lets me insert a unique job only if another job is not in the “pending” state. If a job is already “running,” then I’d want to add a new job, because the running one may be working off stale data.

For me, it’s less about strictly enforcing one-at-a-time processing and more about avoiding unnecessary contention and duplicates. In my case, various API calls update data, and I need to recalculate schedules once those updates are committed. Right now, I can just insert a job for every change, but that risks flooding the queue if a lot of updates come in at once. Ideally, I want only one pending job at a time—until that job actually starts running, at which point a new job should be scheduled if more updates arrive.