feat: internal to MongoDB $jsonSchema conversion COMPASS-8701

[paula-stacho]

While the internal format is very verbose, for $jsonSchema we keep the basics:

• bsonType (converted from the internal bsonType - the map mostly taken from @mcasimir's POC https://github.com/mongodb-js/compass/pull/5690/files)
• required + properties for objects
• items for array
• mixed types are represented either as an array (with non-array non-object types), or with the help of anyOf

[addaleax]

This is synchronous code, so there's no way for an abort to happen between multiple iterations of this loop, right?

[paula-stacho]

Yes, the code is synchronous. I'm not super sure about this abort, the concern was that there might be some extremely nested documents, but most likely that kind of thing would become a problem already at the analysis, so having the abort there is probably enough. What do you think @Anemy ?

[Anemy]

What Anna is getting at is that once this code starts, we're not releasing the javascript thread to do any other work, so we can't have the signal abort except for it already being aborted. To have the functions be cancellable we'll want to yield occasionally, so wrapping these in async and awaiting a promise somewhere inside.

[addaleax]

To have the functions be cancellable we'll want to yield occasionally, so wrapping these in async and awaiting a promise somewhere inside.

Fwiw that still wouldn't end up being interruptible by anything other than microtasks – e.g. a user clicking an "abort" button would still never have any effect

[Anemy]

Would a solution where we occasionally run a set timeout work nicely? Or do you feel we should consider moving this to a worker worker? I feel a worker will add some complexity to this code to something which typically runs pretty quick most of the time, but does sound like the more elegant solution.

if (yieldCounter > 100) {
  yieldCounter = 0;
  await new Promise((resolve) => setTimeout(resolve));
}
yieldCounter++;

[addaleax]

Yeah, worker would be nice, but we'd have to be careful about not breaking webpack in downstream packages or running in browser environments, which is probably where a lot of the complexity would lie

setTimeout without a timeout sounds like a decent approach that's a lot simpler, yeah

[paula-stacho]

Thank you both, turns out I didn't actually understand how this works. I've tried the approach with setTimeout

[addaleax]

Not something for this PR, but do we plan to have integration tests at some point, e.g. ones that take documents, generate schemas for them, then add a validator to a real MongoDB database + try to insert those documents?

[paula-stacho]

Great idea! I was planning something similar in the E2E tests in Compass, but having such integration tests in this package definitely makes sense. I will add a ticket.