Pass@k

[clefourrier]

No description provided.

[clefourrier]

I made it as exhaustive/customizable as the other metrics (full exact match for the individual predictions by default, options to normalize strings in case you use it for math evals for ex) but I can remove some options if you feel that's too much complexity

[clefourrier]

Core logic here

[clefourrier]

Pass at K here, literally the one from codex

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[hynky1999]

Wouldn't be the best to make it dynamic ?
E.g you just wrap an existing metric with it, so that it's more flexible ?
Then I can create it like:
metric=pass_at_k(my_existing_metric, k=10) for example

[clefourrier]

You would be able to create a custom metric like so, with a custom sample level metric:

    your_custom_pass_at = SampleLevelMetric(
        metric_name="pass@",
        sample_level_fn=PassAtK(k=10, sample_scoring_function=my_existing_metric).compute,
        category=MetricCategory.GENERATIVE_SAMPLING,
        use_case=MetricUseCase.REASONING,
        corpus_level_fn=np.mean,
        higher_is_better=True,
    )

unless you need stg else?

[hynky1999]

Ahhh I missed that arg, good by me, thought it was exclusively for string to string comparison