Like most libraries, Bee Queue was made to satisfy a project's need. In this case, the task queue was a potential scaling bottleneck, so the project needed a solution to minimize queue overhead and loosen that constraint. The solution also needed to maintain reliability and fit the actual use cases, which were:

• Receive request -> Enqueue job -> Wait for results -> Send results in response
• Receive request -> Enqueue job -> Send response -> Retrieve results later
• Process n jobs concurrently with one worker process, with network overhead related total jobs processed, not to n

Kue, despite doing everything we needed (and more), ultimately had too much overhead and not enough atomicity. It was not originally designed with the same concerns we had about performance, reliability, and failure recovery.

Bull focused on atomicity, figured out the reliability and failure recovery concerns, and was more lightweight than Kue. Unfortunately, it had too much concurrency overhead, and didn't support our first use case very well, especially with more than one process sending jobs.

Our decision factors included:

• Kue had issues with stuck jobs and duplicate jobs. Though it's been around for a while and has had time to mature, it didn't originally consider atomicity and failure recovery to the same extent that Bull and Bee-Queue have.
• Kue had significant unnecessary Redis and network overhead. Much of it relates to its extra features, but it left a lot of easy optimizations on the table.
• Kue could use more thorough tests.
• Kue and Bull are not laser-focused on minimizing overhead in the way that Bee Queue tries to be.
  • In particular, they both have significant unnecessary overhead with how they handle processing concurrency. The benchmarks reflect this.
• Bull seemed to have some (admittedly very narrow) race condition windows which Bee Queue tries to avoid.
• Bull uses promises throughout. This isn't a bad thing at all, but it's a preference.
• Bull, though lighter than Kue, does still leave some optimizations on the table.
• Bull does not support sending results back with Pub/Sub. This is for understandable reasons, but it makes our first use case very difficult when we have more than one process sending jobs. Beyond that, the downsides to this approach don't really matter to our use case.
• We needed to be able to disable unnecessary connections/events based on what the queue instance needed to do (getEvents, sendEvents, isWorker settings)
• Kue and Bull inline their Lua scripts (they deserve their own files) and do not use SCRIPT LOAD to avoid sending the scripts over the network repeatedly.
• The need to minimize the request-response latency as much as possible. The difference for a single request isn't significant when the server and/or worker are on the same machine as Redis, but as soon you go across the network, the difference can be significant.
  • Most of our jobs run in under a second and we want to return the results immediately in the response, so optimizing for this use case was most important.