You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, we depend on a global variable to manage instances of EngineStatsScraper, RequestStatsMonitor, and Router. We should initialize this variable within the vllm_router.router.InitializeAll function.
To enhance our design, we could implement a singleton pattern for these components. By doing so, we would ensure that only one instance of each class exists throughout the application lifecycle. This approach can help prevent potential issues related to state management and resource usage.
For more information on various methods to implement singletons in Python, you can refer to this Stack Overflow discussion.
The text was updated successfully, but these errors were encountered:
@gaocegege could you assign this to me as well so I can do it next because it's gonna link with what I'm currently doing with the Router Observability (need to modify the EngineStat, RequestStat, etc.) 😄 Seems like we will need some sort of dashboard to keep up who is doing what lol 😆
Currently, we depend on a global variable to manage instances of
EngineStatsScraper
,RequestStatsMonitor
, andRouter
. We should initialize this variable within thevllm_router.router.InitializeAll
function.https://github.com/vllm-project/production-stack/blob/c8cdf9de34b21532089146148efafefbec84d46e/src/vllm_router/router.py#L423:#L431
To enhance our design, we could implement a singleton pattern for these components. By doing so, we would ensure that only one instance of each class exists throughout the application lifecycle. This approach can help prevent potential issues related to state management and resource usage.
For more information on various methods to implement singletons in Python, you can refer to this Stack Overflow discussion.
The text was updated successfully, but these errors were encountered: