Qwen2-vl 推理时，图片分辨率高会报错

[xyf0106]

Reminder

• I have read the README and searched the existing issues.

System Info

• llamafactory version: 0.9.0
• Platform: Linux-5.4.143-2-velinux1-amd64-x86_64-with-glibc2.35
• Python version: 3.10.15
• PyTorch version: 2.4.1+cu121 (GPU)
• Transformers version: 4.45.2
• Datasets version: 2.19.1
• Accelerate version: 0.34.2
• PEFT version: 0.12.0
• TRL version: 0.9.6
• GPU type: NVIDIA A100-SXM4-80GB
• DeepSpeed version: 0.15.2

Reproduction

FORCE_TORCHRUN=1 CUDA_VISIBLE_DEVICES=0 llamafactory-cli train examples/inference/Qwen2_vl_lora_sft.yaml

Qwen2_vl_lora_sft.yaml 内容如下

`model_name_or_path: Qwen/Qwen2-VL-7B-Instruct/
adapter_name_or_path: saves/Qwen2VL-7B-Instruct/lora/train_2024-10-21-v2/checkpoint-2000/

method

stage: sft
do_predict: true
finetuning_type: lora

dataset

eval_dataset: item_satisfy_data_1001_temp
template: qwen2_vl
cutoff_len: 1024
max_samples: 2
overwrite_cache: true
preprocessing_num_workers: 1

output

output_dir: /mnt/nas/xuyufan/LLaMA-Factory/output/Qwen2-VL-7B-Instruct/train_2024-10-21/checkpoint-2000
overwrite_output_dir: true

eval

per_device_eval_batch_size: 1
predict_with_generate: true
ddp_timeout: 180000000
`

Expected behavior

用finetune后的模型完成推理

Others

有两个问题：

1. 一个是图片size大的时候，推理会报错：RuntimeError: shape mismatch: value tensor of shape [3, 1049] cannot be broadcast to indexing result of shape [3, 1007]
2. 还有一个是，max_samples超过几百之后，会卡在切词这里：Running tokenizer on dataset

[hiyouga]

增加 cutoff_len