Make mongodb a peer dependency

[sebamarynissen]

This PR is based on my fork and provides support for mongodb@6 by making the mongodb package a peer dependency. Version 4 and 5 of the drivers are still supported too.

One of the benefits of making mongodb a peer dependency is that there is no longer a mismatch between ObjectId of different mongodb versions as agenda will use the ObjectId() from the same mongodb version that you specified during construction. I've experienced that this could sometimes lead to errors.

I've also changed the module to become esm only to prepare for any dependencies becoming esm only one day too.

[Apidcloud]

Agenda would really benefit from this peer dependency. Atm we have to convert from one ObjectId to another because it's not.

[bompus]

I support this PR over my own ( #46 ), as it ensures there isn't a mismatch between mongodb driver versions.

This not only provides faster start-up times by not loading in two different driver versions, but is likely to eliminate some bugs that could arise from running different versions, such as the ObjectId logic already mentioned.

[Apidcloud]

MongoDB 5.0 will reach end of life this month (October 2024), and providers like Digital Ocean are already dropping support to 5.0. Any chance we can get this merged asap?

Edit: Any chance you could publish your fork to npm meanwhile, @sebamarynissen ? Ah nvm. Just assumed it wasn't published because of the README, but looked at the package.json and it's available through @whisthub/agenda on npm.

[sebamarynissen]

My fork is available on npm as @whisthub/agenda. I've been using it in production for a while without any issues.

As for merging this PR, unfortunately I fear that the repo has been effectively abandoned, but my goal is to keep my fork up to date with the latest version of the mongodb driver.

[Apidcloud]

Ah, yes. I found it a bit ago through the package.json. I'll try to switch to it, then. Thanks @sebamarynissen

[Apidcloud]

How hard would it be to output commonjs instead of esm @sebamarynissen ? Unfortunately the Digital Ocean thing caught us by surprise and we don't have much time to update our codebases to ESM. I have tried node 22 with --experimental-detect-module and dynamic import your fork, but still couldn't get it to work

[sebamarynissen]

I think the flag you're looking for is --experimental-require-module, though I would advise to just use a dynamic import. agenda.start() is asynchronous anyway, so normally it shouldn't require a major refactor of your codebase to use a dynamic import in cjs.

[Apidcloud]

Thanks! That's indeed the flag I was looking for. The dynamic import doesn't seem to work with SWC. I'll give that a go, and report here later

[Apidcloud]

The flag --experimental-require-module seems to have done the trick with Node 22.

Are you using MongoDb 7 (or the just released v8) in production @sebamarynissen ? We might as well upgrade directly to at least 7 to avoid having to do this again in a few months.

[sebamarynissen]

I'm using MongoDB 7 in production, but that shouldn't really matter as long as you use the latest mongodb@6 node.js driver, as it's both compatible with MongoDB 6, 7 and 8, see https://www.mongodb.com/docs/drivers/node/current/compatibility/.

[Apidcloud]

@sebamarynissen, which mongodb indices do you use for Agenda?
From the point we started using Agenda like 2 years ago, we had to create one that is slightly different that the recommended findAndLockNextJobIndex, as it wasn't picked up at all:

After upgrading in October 2024 to the version in whisthub agenda, that was still the case, and pretty much everything was well. But today the CPU suddenly went to 100% with the agenda queries being very slow. We literally had no changes anywhere in the codebase. There almost no writes or anything. ops maybe a bit higher than usual but also nothing special I would say.

Since October 2024, the findAndLockNextJobIndex had been used 14 times. When I was digging into it today, I went to the whisthub agenda codebase and created one new index to tackle this conditional (https://github.com/whisthub/agenda/blob/93d1348fd5fc854fd8e0866e36bb46cf8961b7d9/src/JobDbRepository.ts#L133):

The new index got 0 uses. But, somehow, after adding it, the one with 14 (findAndLockNextJobIndex) immediately started getting used (??). The load was still close to 100% so I think that didn't change anything for an hour or two. After a while, I removed the new index with 0 uses, since, well, it wasn't used and apparently not helping. To my surprise findAndLockNextJobIndex keeps getting used and is now at 18594 hits (from originally 14). The first index in the picture (name_1_disabled_1_lockedAt_1_nextRunAt_1) is still getting picked up, so I'm not sure what's going on--just glad it seems to be ok for now.

For reference, the slowest queries, as per mongo compass performance tab, are REMOVE table-agenda. They ranged from 1s to 7s. Now that I'm thinking about it maybe I'm looking for the wrong thing in agenda's codebase?