Only get Lower accuracy than reported in the paper

[justgogorunrun]

Is anyone evaluating pre-trained parametric models with much lower accuracy than reported in the paper? What are the prompts you are using in your evaluations? When I set it to take 8 frames, the accuracy was only 44, almost 4 less than in the paper.

[kcz358]

May I ask which dataset you are evaluating with? I evaluated videomme recently and the score seems fine. We do notice that with the upgrade of the lmms-eval, the score changes a bit but in most of the cases the scores are increasing