feat: Offline batched inference based on OpenAI offline batching API

[gaocegege]

This issue is created to track the conversation about the offline batch API.

gaocegege@

I was just wondering if there are any plans to add openai api compatible offline batch support in the router. I saw a comment about it vllm-project/vllm#1636 (comment), and it looks like this feature needs file interfaces for uploads, which might not fit well with vLLM. It could be a nice addition to the router, acting as a bridge between users and vLLM.

ApostaC@

Seems like there should be multiple design choices to handle file uploads. Will add this to the roadmap (which will be released as an issue in the project tomorrow). Maybe we can have more detailed discussion there?

simon-mo@

I also agree this can be a lightweight optional component in the stack! given in K8s you can provision persistent volume easily or mount s3-fuse

[gaocegege]

We could start by keeping the files in the local file system, especially since our router currently doesn't support multi-instance deployment. We can design a robust abstraction for file uploads to ensure extensibility, allowing us to support other storage backends (e.g., MinIO, S3) in the future, particularly for k8s environments.

[ApostaC]

@gaocegege Thanks for leading the discussion here!

We could start by keeping the files in the local file system, especially since our router currently doesn't support multi-instance deployment.

The proposal looks good to me! Would you like to take a stab at designing the concrete interface in the router?

[gaocegege]

Yes! I'd be happy