Failure to reproduce AIME score on Sky-T1-32B-Preview

[RyanMarten]

python eval.py --model NovaSky-AI/Sky-T1-32B-Preview --evals=AIME --tp=8 --output_file=results.txt --temperatures 0.7

{"acc": 0.3667}

The reported score was 43.3, which is significantly different.

[RyanMarten]

Looks like @fanqiwan similarly got a lower AIME score:

#33 (comment)

[2proveit]

python eval.py --model NovaSky-AI/Sky-T1-32B-Preview --evals=AIME --tp=8 --output_file=results.txt --temperatures 0.7

{"acc": 0.3667}

The reported score was 43.3, which is significantly different.

my score is similar to yours, 0.333

[rucnyz]

python eval.py --model NovaSky-AI/Sky-T1-32B-Preview --evals=AIME --tp=8 --output_file=results.txt --temperatures 0.7

{"acc": 0.3667}
The reported score was 43.3, which is significantly different.

my score is similar to yours, 0.333

my score is even worse, 0.30

[tyler-griggs]

Hi all, thanks for calling out these issues! We have also discovered some issues in our evaluation framework and are currently working to organize and refactor it at #47.