Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Implement singleton pattern for enginestatscraper, requeststatsmonitor, and router #86

Open
gaocegege opened this issue Feb 8, 2025 · 1 comment
Assignees
Labels
good first issue Good for newcomers help wanted Extra attention is needed

Comments

@gaocegege
Copy link
Collaborator

gaocegege commented Feb 8, 2025

Currently, we depend on a global variable to manage instances of EngineStatsScraper, RequestStatsMonitor, and Router. We should initialize this variable within the vllm_router.router.InitializeAll function.

https://github.com/vllm-project/production-stack/blob/c8cdf9de34b21532089146148efafefbec84d46e/src/vllm_router/router.py#L423:#L431

To enhance our design, we could implement a singleton pattern for these components. By doing so, we would ensure that only one instance of each class exists throughout the application lifecycle. This approach can help prevent potential issues related to state management and resource usage.

For more information on various methods to implement singletons in Python, you can refer to this Stack Overflow discussion.

@gaocegege gaocegege added good first issue Good for newcomers help wanted Extra attention is needed labels Feb 8, 2025
@sitloboi2012
Copy link
Contributor

@gaocegege could you assign this to me as well so I can do it next because it's gonna link with what I'm currently doing with the Router Observability (need to modify the EngineStat, RequestStat, etc.) 😄 Seems like we will need some sort of dashboard to keep up who is doing what lol 😆

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants