bug: CI functionality test needs to build latest docker image

[Shaoting-Feng]

Describe the bug

CI functionality test pipeline currently uses remote router image. If some contributors change the codes with the router, the functionality test will not be able to detect whether there is an error or not.

To Reproduce

None

Expected behavior

No response

Additional context

The functionality test should build router docker image as an initial step.

[gaocegege]

Currently, our Docker image action repushes the image with the same tag if the router version in setup.py remains unchanged, which is not a common practice, The versioned image should be immutable.

https://github.com/vllm-project/production-stack/blob/main/.github/workflows/router-docker-release.yml#L45

I suggest using the Git commit as the tag for the pull request, and only incrementing the version during the release process.

We could build the image using the Git commit, run the action initially, and then proceed with the end-to-end tests.

[Shaoting-Feng]

I am thinking for pull requests and commits, we can build the router image and push it to a port on the local GitHub runner machine instead of pushing it to the GitHub repository. The workflow can then pull the Docker image from localhost. This approach avoids saving potentially wrong image on github.

[Shaoting-Feng]

For your quotes, if we only run this Docker release pipeline when merging into the main branch, it means the PR has already passed the CI pipelines. Using this router as the "latest" image allows users to try the new features without bugs.

[gaocegege]

"Latest" works for me, but my main concern is about the versioned image. It goes against the expectation that the version is pinned.

[gaocegege]

I am thinking for pull requests and commits, we can build the router image and push it to a port on the local GitHub runner machine instead of pushing it to the GitHub repository. The workflow can then pull the Docker image from localhost. This approach avoids saving potentially wrong image on github.

Maybe we could use artifacts. https://docs.docker.com/build/ci/github-actions/share-image-jobs/

[Shaoting-Feng]

"Latest" works for me, but my main concern is about the versioned image. It goes against the expectation that the version is pinned.

I agree. We should update the current "build router image" pipeline.

[Shaoting-Feng]

I am thinking for pull requests and commits, we can build the router image and push it to a port on the local GitHub runner machine instead of pushing it to the GitHub repository. The workflow can then pull the Docker image from localhost. This approach avoids saving potentially wrong image on github.

Maybe we could use artifacts. https://docs.docker.com/build/ci/github-actions/share-image-jobs/

Thanks. However, I'm not sure if Minikube can pull images from Artifacts.

I've tested using the local Docker registry, and it works locally. I'll update the "Functionality Test" pipeline accordingly.