Support for Mongodb Enterprise version and "inMemory" storage engine

[nickrobillard]

As mentioned here and here, storage engine ephemeralForTest has been EOL'ed and was completely removed starting in Mongodb 7.0. Also as mentioned, only Mongodb Enterprise - not Community Edition - supports a newer storage engine called inMemory. I have done some tests with it, and while starting and stopping the db seems slower than ephemeralForTest, it seems much faster than wiredTiger for my specific use case (~300 tests with db stop and start in between most of them). Storage engine inMemory is also officially documented - here and here - unlike ephemeralForTest and it seems to have a long-term future. There are other reasons to support Mongodb Enterprise for other Enterprise-only features like Client-Side Field Level Encryption.

I have done some initial work and tests to support Mongodb Enterprise in MMS. It is quite simple to basically support it. But I'd like to get some feedback on concept and the approach.

Working assumptions and constraints

• ephemeralForTest is end of life'd and has been completely removed starting in Mongodb 7.0.
• In some cases, ephemeralForTest appears to be less stable in Mongodb 5.x than it was in 4.x.
• inMemory storage engine is only supported in Mongodb Enterprise and it appears to be the long term successor to ephemeralForTest.

How to support it

My first approach was to simply support Enterprise tgz urls via downloadUrl config. I got it working in a very crude way after making a minor change to the regex parsing logic (to ignore the -enterprise path part). util/DryMongoBinary.js regex change:

/mongodb-(?<platform>linux|win32|osx|macos)(?:-ssl-|-)(?<arch>\w{4,})(?:-enterprise|)(?:-(?<dist>\w+)|)(?:-ssl-|-)(?:v|)(?<version>[\d.]+(?:-latest|))\./

With this change, I can use this in package.json to use Mongodb Enterprise 6.0.20 and run the default ephemeralForTest engine:

"config": {
  "mongodbMemoryServer": {
    "downloadUrl": "https://downloads.mongodb.com/osx/mongodb-macos-arm64-enterprise-6.0.20.tgz"
  }
}

To use the new inMemory engine, I just pass this config into MMS create method:

const mongoServer = await MongoMemoryServer.create({
  instance: {
    storageEngine: 'inMemory'
  }
});

How to make it configurable?

Obviously, we want MMS to automatically download the correct tgz file based on system OS and CPU arch. But how should it work? Should MMS users simply be able to configure a boolean like useEnterprise=true|false (default false)? How should users toggle between storage engines? I know that recent versions of MMS automatically switch to wiredTiger when Mongodb version 7.0+ is used. Also, instead of having to set the storage engine when instantiating MMS, I think it would be more ergonomic to also be able to configure the default storage engine via env var or package.json.

Considerations & complications

• Mongodb Enterprise version is not free, but is available to download and use for evaluation and development purposes. Will this impact MMS's desire to support it?
• Hardcoding the url patterns assumes that the tgz urls will remain stable. Considering that the user experience to download Enterprise here https://www.mongodb.com/try/download/enterprise is different than CE, and requires user info to be entered before download starts, maybe Mongodb does not want it to be downloaded in an automated way. I don't know.
• A question that needs to be asked - is this even a direction that MMS wants to go?

References

I will also mention this issue because it was the first one I came across when I looked for Mongodb Enterprise support - #128

[hasezoey]

Thanks for this write-up and testing.

A question that needs to be asked - is this even a direction that MMS wants to go?

I dont know if it will be fully supported, but i would not be against allowing the download url binary name and adding the storage-engine to the list (with explicit call-out about enterprise-only).

Hardcoding the url patterns assumes that the tgz urls will remain stable.

From what i can tell, at least currently, the enterprise urls are basically the same as the community versions aside from the added -enterprise- in the name.

Mongodb Enterprise version is not free, but is available to download and use for evaluation and development purposes. Will this impact MMS's desire to support it?

I have no clue about legal things. In the download you provided, downloading there seemingly wants you to give some information and agree to the Customer Agreement EULA, though the Archive downloads seemingly dont require any kind of information sharing / EULA agreement (like the community editions).
As i am not clear on this, i will hold-off to officially support selecting & downloading it like the community edition via a flag like useEnterprise.

EDIT: nevermind, i didnt read the first sentences of the archive page:

Please note that for those who are not yet MongoDB customers, download and use constitutes acceptance of the Customer Agreement. For non-Enterprise MongoDB releases, click here.

Though this implies that it is basically the same regardless if it is community or enterprise. (For reference, the community archive)