Performance issues.

[dasveloper]

I have noticed a large performance hit when implementing mongo-cursor-pagination and I'm trying to determine if that is expected and if there is anything that can be improved. I'm using the Mongoose implementation set up like this:

const mongoose = require('mongoose');
const MongoPaging = require('mongo-cursor-pagination');

const logSchema = new mongoose.Schema({
  bucket: { type: mongoose.Schema.Types.ObjectId, ref: 'Bucket' },
  message: String,
  type: Number,
}, { timestamps: true });

logSchema.plugin(MongoPaging.mongoosePlugin);
logSchema.index({ createdAt: 1 });
logSchema.index({ updatedAt: 1 });
logSchema.index({ _id: 1 });

const Log = mongoose.model('Log', logSchema);
module.exports = Log;

I have created 300,000 test records.

As a control, I used a standard .find() call with no query and limited to 20 results.

console.time('test');
const logs = await Log.find({}).limit(20);
console.timeEnd('test');

Running it 3 times I got an average ~30ms response.

test: 32.08ms
test: 34.162ms
test: 29.232ms

Next I tried using .paginate based on createdAt (which is indexed), still with no query and same 20 limit.

console.time('test');
const logs = await Log.paginate({
    query: {},
    paginatedField: 'createdAt',
     limit: 20,
});
console.timeEnd('test');

Running it 3 times you can see I got an average ~500ms response, over 10x higher.

test: 571.058ms
test: 491.583ms
test: 483.037ms

Next I tried used the same .paginate query and limit, but added the next token returned from the previous response.

console.time('test');
const logs = await Log.paginate({
    query: {},
    paginatedField: 'createdAt',
    limit: 20,
    next: 'W3siJGRhdGUiOiIyMDIxLTA2LTE4VDE2OjQ2OjE5LjgxOFoifSx7IiRvaWQiOiI2MGNjY2RkYmUxZjZhMWM3NDZiZDYzMzUifV0',
});
console.timeEnd('test');

Running it 3 times I now got an average ~900ms response, almost 2x higher.

test: 878.252ms
test: 945.973ms
test: 854.034ms

Luckily it doesn't appear to get worse any further down the line, so it still more performant than a limit/offset style pagination. But it just seems odd to me that it is so much slower than a normal .find() when no extra querying is added and uses the same limit and indexed field. Plus it seems odd that adding a next param again dramatically increases response.

[bradvogel]

There's likely a missing index on the collection. Can you turn on Mongo query logging and report the exact Mongo queries that are run by this library? That will inform what index should be created.