Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix] Clamp logprob with dtype min to prevent -inf #3224

Merged
merged 4 commits into from
Jan 31, 2025

Conversation

ByronHsu
Copy link
Collaborator

@ByronHsu ByronHsu commented Jan 30, 2025

Motivation

To fix #2955
Follow up #3212

The probs can have 0 and after torch.log it becomes -inf, making fastapi not able to construct response #2955.

This PR adds a clamp (https://pytorch.org/docs/stable/type_info.html) based on dtype to prevent -inf.

Modifications

Checklist

Sorry, something went wrong.

@ByronHsu ByronHsu changed the title [test] add a small eps to prevent log(0) for return_logprob [WIP] [test] add a small eps to prevent log(0) for return_logprob Jan 30, 2025
@ByronHsu ByronHsu changed the title [WIP] [test] add a small eps to prevent log(0) for return_logprob [bug] Clamp logprob with dtype min to prevent -inf Jan 30, 2025
@zhyncs zhyncs changed the title [bug] Clamp logprob with dtype min to prevent -inf [fix] Clamp logprob with dtype min to prevent -inf Jan 31, 2025
@zhyncs zhyncs merged commit 734daed into main Jan 31, 2025
17 checks passed
@zhyncs zhyncs deleted the byhsu/solve-inf-logprob branch January 31, 2025 09:04
chongli-uw pushed a commit to chongli-uw/sglang that referenced this pull request Feb 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug] JSONResponse fails if the probability distribution is very spiky.
2 participants