Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Misc] Support register quantization method out-of-tree #11969

Merged
merged 2 commits into from
Jan 19, 2025
Merged

[Misc] Support register quantization method out-of-tree #11969

merged 2 commits into from
Jan 19, 2025

Conversation

ice-tong
Copy link
Contributor

@ice-tong ice-tong commented Jan 12, 2025 •
edited by github-actions bot
Loading

Motivation

@github-actions GitHub Actions
Copy link

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

  • Add ready label to the PR
  • Enable auto-merge.

🚀

@ice-tong ice-tong changed the title [Feat] Support register quantization method out-of-tree [Misc] Support register quantization method out-of-tree Jan 12, 2025
@DarkLight1337 DarkLight1337 requested a review from mgoin January 12, 2025 14:44
@ice-tong
Copy link
Contributor Author

@DarkLight1337 @mgoin Hey there! Just a quick reminder about the pull request I submitted a few days ago.

We’re looking forward to vllm supporting the registration of quantization methods from external sources, which will help us develop quantization tools for evaluating fake-quantization models.

I’d really appreciate your feedback on this PR whenever you have a moment! Thank you!

mgoin
mgoin reviewed Jan 17, 2025
View reviewed changes
Copy link
Member

@mgoin mgoin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be a great feature! Could you add a test for adding a simple quantization method? It could be a trivial quant-dequant pass

@ice-tong
Copy link
Contributor Author

Hi @mgoin, thank you very much for taking the time to review this PR!

I've added a test as suggested, which is used to register a custom per-token dynamic fake-quant method.

If you have any questions or suggestions for further modifications, please feel free to let me know.

@ice-tong ice-tong force-pushed the yancong_register_quantization_config branch from 8bf911c to 56b129b Compare January 18, 2025 07:04
@mergify mergify bot added documentation Improvements or additions to documentation ci/build labels Jan 18, 2025
@mergify Mergify
Copy link

mergify bot commented Jan 18, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ice-tong.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@ice-tong ice-tong force-pushed the yancong_register_quantization_config branch from 83290eb to 15ebfd0 Compare January 18, 2025 07:18
mgoin
mgoin approved these changes Jan 18, 2025
View reviewed changes
Copy link
Member

@mgoin mgoin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent work, I appreciate it!

@mgoin mgoin added the ready ONLY add when PR is ready to merge/full CI is needed label Jan 18, 2025
@mgoin mgoin enabled auto-merge (squash) January 18, 2025 18:21
@simon-mo simon-mo merged commit 32eb0da into vllm-project:main Jan 19, 2025
66 of 68 checks passed
joennlae pushed a commit to 44ai-labs/vllm that referenced this pull request Jan 19, 2025
@ice-tong
[Misc] Support register quantization method out-of-tree (vllm-project…
02e730b 
…#11969)
joennlae pushed a commit to 44ai-labs/vllm that referenced this pull request Jan 19, 2025
@ice-tong
[Misc] Support register quantization method out-of-tree (vllm-project…
150e6c8 
…#11969)
lckr pushed a commit to lckr/vllm that referenced this pull request Jan 19, 2025
@ice-tong
[Misc] Support register quantization method out-of-tree (vllm-project…
2175385 
…#11969)
@wangxiyuan wangxiyuan mentioned this pull request Jan 20, 2025
Merged
@ice-tong ice-tong mentioned this pull request Jan 21, 2025
Open
@kzawora-intel kzawora-intel mentioned this pull request Jan 21, 2025
Open
abmfy pushed a commit to abmfy/vllm-flashinfer that referenced this pull request Jan 24, 2025
@ice-tong @abmfy
[Misc] Support register quantization method out-of-tree (vllm-project…
3a73b44 
…#11969)

Signed-off-by: Bowen Wang <[email protected]>
abmfy pushed a commit to abmfy/vllm-flashinfer that referenced this pull request Jan 24, 2025
@ice-tong @abmfy
[Misc] Support register quantization method out-of-tree (vllm-project…
278b145 
…#11969)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mgoin mgoin mgoin approved these changes

@comaniac comaniac Awaiting requested review from comaniac

@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth tlrmchlsmth is a code owner

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon

@DarkLight1337 DarkLight1337 Awaiting requested review from DarkLight1337

@ywang96 ywang96 Awaiting requested review from ywang96

@simon-mo simon-mo Awaiting requested review from simon-mo

@njhill njhill Awaiting requested review from njhill

@zhuohan123 zhuohan123 Awaiting requested review from zhuohan123

@youkaichao youkaichao Awaiting requested review from youkaichao

Assignees
No one assigned
Labels
ci/build documentation Improvements or additions to documentation frontend ready ONLY add when PR is ready to merge/full CI is needed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feature]: Support out-of-tree register of customize quantization method
3 participants
@ice-tong @mgoin @simon-mo