remote repo main 브랜치 업데이트 적용 #8
Merged
+2,914
−104
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* add charades_sta for temporal grounding * update * update formatting --------- Co-authored-by: john.zhang <[email protected]>
* Update README.md * Update README.md * [Fix] of "mvbench missing videos" Modify DATA_LIST so that it can find the corresponding video * [Fix] of "evaluation of llava_vid on mvbench" "mvbench_video/tvqa/frames_fps3_hq/castle_s07e04_seg02_clip_14" is the sampled video frame, not the original video file. And The current code logic cannot handle this subtask. * Fixing lmms_eval/models/llava_vid.py
* Add VLLM model integration and update configurations - Introduce VLLM model in the model registry. - Update AVAILABLE_MODELS to include new models: - models/__init__.py: Added "aria", "internvideo2", "llama_vision", "oryx", "ross", "slime", "videochat2", "vllm", "xcomposer2_4KHD", "xcomposer2d5". - Create vllm.py for VLLM model implementation: - Implemented encoding for images and videos. - Added methods for generating responses and handling multi-round generation. - Update mmu tasks with new prompt formats and evaluation metrics: - mmmu_val.yaml: Added specific kwargs for prompt types. - mmmu_val_reasoning.yaml: Enhanced prompts for reasoning tasks. - utils.py: Adjusted evaluation rules and scoring for predictions. - Added script for easy model execution: - vllm_qwen2vl.sh: Script to run VLLM with specified parameters. * Set environment variables for VLLM script - Configure environment for better performance and debugging. - Added variables to control multiprocessing and NCCL behavior. miscs/vllm_qwen2vl.sh: - Set `VLLM_WORKER_MULTIPROC_METHOD` to `spawn` for compatibility. - Enabled `NCCL_BLOCKING_WAIT` to avoid hangs. - Increased `NCCL_TIMEOUT` to 18000000 for long-running processes. - Set `NCCL_DEBUG` to `DEBUG` for detailed logs. * Rename scripts and update paths - Renamed representation scripts for clarity. - miscs/repr_scripts.sh -> miscs/model_dryruns/llava_1_5.sh - miscs/cicd_qwen2vl.sh -> miscs/model_dryruns/qwen2vl.sh - miscs/tinyllava_repr_scripts.sh -> miscs/model_dryruns/tinyllava.sh - miscs/vllm_qwen2vl.sh -> miscs/model_dryruns/vllm_qwen2vl.sh - Updated parameters in the vllm_qwen2vl.sh script. - miscs/model_dryruns/vllm_qwen2vl.sh: Added `--limit=64` to output path command. * Optimize image handling in VLLM model - Simplify image conversion in the `to_base64` method: - vllm.py: Directly convert input image to RGB format instead of copying it. - Remove unnecessary base64 encoding for images: - vllm.py: Return the PIL image directly instead of converting it to base64. - Update video frame processing to return PIL images: - vllm.py: Replace base64 encoding of frames with returning the PIL frames directly. * Revert "Optimize image handling in VLLM model" This reverts commit 469e1fc. * use threads to encode visuals --------- Co-authored-by: kcz358 <[email protected]>
* Update API usage and add environment configuration - Adjust API client initialization for OpenAI and Azure. - Modify image encoding to handle size limits and maintain aspect ratio. - Add new `OpenAICompatible` model class. - Introduce retry mechanism for API requests with configurable retries. - Update `.gitignore` to exclude `.env` and `scripts/`. .gitignore: - Exclude `.env` file for security. - Ensure no scripts directory is tracked. lmms_eval/models/gpt4v.py: - Refactor API client initialization to use new OpenAI and Azure clients. - Update image encoding to handle size limits with resizing logic. - Adjust retry logic for API calls, reducing sleep time. lmms_eval/models/openai_compatible.py: - Create new `OpenAICompatible` model class with similar structure. - Implement encoding functions for images and videos. - Integrate environment variable loading and persistent response caching. miscs/model_dryruns/openai_compatible.sh: - Add sample script for running the new model. * Improve code readability and organization - Remove unused import for deepcopy in `openai_compatible.py`. - Add a blank line for better separation of code sections. - Adjust comment formatting for `max_size_in_mb` for consistency. - Ensure consistent spacing around comments. File: `lmms_eval/models/openai_compatible.py` - Removed `deepcopy` import: cleaned up unnecessary code. - Added blank line after `load_dotenv`: improved readability. - Reformatted comment on `max_size_in_mb`: enhanced clarity. - Removed extra blank line before `Accelerator`: tightened spacing. * Fix init --------- Co-authored-by: kcz358 <[email protected]>
* [Model] add Ola audio * [Model] add Ola * [Model] Ola * [Model] add Ola * [Model] add Ola * [Model] add Ola * [Model] add Ola * Direct import for moviepy --------- Co-authored-by: kcz358 <[email protected]>
* Update utils.py * Update utils.py * Update utils.py
…nk (EvolvingLMMs-Lab#560) * upload egogpt model * support egoplan benchmark for lmms-eval * support egothink benchmark for lmms-eval
tkdcjf159
approved these changes
Mar 4, 2025
|
|
||
| # If image is too large, resize it while maintaining aspect ratio | ||
| while len(byte_data) > max_size and img.size[0] > 100 and img.size[1] > 100: | ||
| new_size = (int(img.size[0] * 0.75), int(img.size[1] * 0.75)) |
There was a problem hiding this comment.
원본의 75%로 리사이즈 하는군요. 리사이즈 해도 엄청 큰 이미지도 있을텐데 ㅎㅎ 체크하고 갑니다.
| self.threads = threads | ||
|
|
||
| accelerator = Accelerator() | ||
| self.client = LLM( |
There was a problem hiding this comment.
vllm 에서 동작하게 만들면 lmms-eval에서 쓸수 있겠군요.
There was a problem hiding this comment.
네, 지금 eval 돌리면 40-50분 걸리는데, 좀 더 빨라지지 않을까 합니다.
필수인 듯 하네요 ㅋ
| @@ -16,11 +26,15 @@ metric_list: | |||
| lmms_eval_specific_kwargs: | |||
| default: | |||
| prompt_type: "reasoning" | |||
| multiple_choice_prompt: "Please show step-by-step reasoning, and answer the question with option letter from given choices." | |||
| open_ended_prompt: "Please show step-by-step reasoning, and answer the question using a single word or phrase." | |||
| multiple_choice_prompt: "Please reason step by step, and answer the question with option letter from given choices in the format of Answer: <option letter>." | |||
| export NCCL_TIMEOUT=18000000 | ||
| export NCCL_DEBUG=DEBUG | ||
|
|
||
| python3 -m lmms_eval \ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
내용
https://github.com/EvolvingLMMs-Lab/lmms-eval 의 메인 브랜치에 업데이트 된 사항들을 적용합니다.
DocVision과는 관련 없는 부분이라서 그냥 한번 쭉 봐주신 후에 approve 해주셔도 될 것 같습니다.