[Core] Create metrics for external cache services #3
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
happyandslow
changed the title
happyandslow
force-pushed
the
lexu/vineyard-metrics
branch
from
e3eee41 to
e032a48
Compare
Based on another version of vllm: sighingnow@d347dab Cherry-pick from commit d347dab Signed-off-by: Tao He <[email protected]> (cherry picked from commit 1545f6bf7edcd667e305d3fbcadd913066f04747) resolving vllm update diff temporarily comment out torch.distributed for single node env add VineyardCacheConfig with https://github.com/v6d-io/v6d/blob/ebe8f077e3d3780a27d49238c501854b6b8e29df/modules/llm-cache/ds/kv_cache_block.cc#L163 commented out; cache_ops fix remove CacheConfig from argument (configure through ENV) v6d: fix integration w/ v1 APIs Signed-off-by: Haiyang Shi <[email protected]> Change model_runner to latest version cherry pick model_runner from d347dab source sighingnow@d347dab fix reshape_and_cache_flash argument add cache prefetch/update to work_base clean up Fix after rebase to 029c71d remove tensor copy from cache managed address to pin memory clean up Add fixes to address comments adding cache service metrics initial adding cache service metrics initial update ttft metrics update prefix caching with max num seqs argument
happyandslow
force-pushed
the
lexu/vineyard-metrics
branch
from
e032a48 to
306aa72
Compare
Jeffwan
reviewed
Oct 2, 2024
| @@ -26,6 +26,8 @@ | |||
| from vllm.usage.usage_lib import UsageContext | |||
| from vllm.utils import Counter, deprecate_kwargs, is_list_of | |||
|
|
|||
| import numpy as np | |||
There was a problem hiding this comment.
It's best not to import all names from a module, but just those names that you actually use. In other words, you should try to always use the from ... import ... form of imports.
Also, this import line probably should go with other "system"/"thirdparty" imports before the vllm-specific imports.
| @@ -734,10 +740,29 @@ def _run_engine( | |||
| f"est. speed input: {in_spd:.2f} toks/s, " | |||
| f"output: {out_spd:.2f} toks/s") | |||
| pbar.update(1) | |||
| if self.llm_engine.cache_service_metrics is not None: | |||
There was a problem hiding this comment.
The metrics calculation and logging logic should go inside the individual metrics class instead of embedded into the vLLM engine high-level event-loop:
- the engine logic should encodes abstract operations such as collecting one measurement per step, or finish the measurements (and trigger all derived metrics calculation) once a request is finished.
- the individual metrics class will handle its own logic of which measurement to track and derived metrics to calculate and log.
This way, the engine code will remain stable while we plug in different metrics implementations
Jeffwan
reviewed
Oct 24, 2024
| @@ -26,6 +26,8 @@ | |||
| from vllm.usage.usage_lib import UsageContext | |||
| from vllm.utils import Counter, deprecate_kwargs, is_list_of | |||
|
|
|||
| import numpy as np | |||
Jeffwan
pushed a commit
that referenced
this pull request
Nov 17, 2024
Optimize the KV transfer pipe implementation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.