Config options to speed up inference for ce-esci-MiniLM-L12-v2

[megh-khaire]

Hello, we've deployed metarank with ce-esci-MiniLM-L12-v2 for reranking.
Wanted to know if there are any configuration options available that would speed up the inference process.
Our use case requires the reranking to be as fast as possible...

Here is my current config:

inference:
  msmarco:
    type: cross-encoder
    model: metarank/ce-esci-MiniLM-L12-v2

The batch size is 20 (fixed) with small documents restricted to 255 chars.
We are deploying this on a CPU, with 16GB RAM.