Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DOC: Defining evaluation "success" #654

Open
angadn opened this issue Jan 28, 2025 · 0 comments
Open

DOC: Defining evaluation "success" #654

angadn opened this issue Jan 28, 2025 · 0 comments

Comments

@angadn
Copy link

angadn commented Jan 28, 2025

This quickstart now shows that evaluator callables can return a dict, optionally naming the metric with the key field. That's great, and saves me the weird fn.__name__ override I was doing previously

However, I notice the UI shows a success badge, and I'm not sure what it means - does this mean no exceptions were thrown, or am I able to somehow define thresholds for each metric to consider an evaluation as a success?

Is this somehow possible with the dict too? I'd expect evaluate/aevaluate to take some kind of lambda that sets the success field, but can't find something to this effect

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant