-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Performant Priority Queues #1866
Comments
The original implementation of priority queues used a queue for every priority, but it gets messy and difficult to avoid all special cases. The current implementation is much more solid and simple so thats a really good thing. |
Would |
@zebulonj they are not good enough because they just pop the element from the set, so if the guy popping the element dies for some reason, that job will be in a undefined state. We would need a command like "BZPOPMINLPUSH", that pops and atomically pushes the element to a list, then you can safely move a job from priority list to active list. |
but if we do not care about blocking then it is trivial, we can just have a command that consumes from ZSET and pushes in LIST atomically using lua. |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
this is better handled since bullmq v4 https://github.com/taskforcesh/bullmq/blob/master/docs/gitbook/changelogs/changelog-v4.md#400-2023-06-21 where insertions is O(log(N)) |
I've been using Bull for more than a year in a large-scale, high-performance application (1000+ jobs per minute, 24/7)... and am very happy with it. On the horizon I have the need to add priority to our queues, but O(N) performance per insert is likely going to pose a problem (I expect to be dumping upwards of a million jobs at a time into queues on a regular basis). Opening this issue because I'm interested in learning whether there has been any discussion of more performant architectures for handling priority via Bull queues. Off the top of my head, I'd be interested in running three or four fixed priority queues (where each queue represented jobs of the same priority), with the set of queues feeding into workers that each pull from the highest priority queue with a pending job... thereby removing the need to sort jobs within a queue based on priority.
I'm happy to toss around ideas and ultimately contribute code. Just thought I'd raise this with the community before I invest in any particular path.
The text was updated successfully, but these errors were encountered: