Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ch_PP-OCRv4_det_server fine-tune failed #14633

Open
3 tasks done
Kyo1234567 opened this issue Feb 7, 2025 · 2 comments
Open
3 tasks done

ch_PP-OCRv4_det_server fine-tune failed #14633

Kyo1234567 opened this issue Feb 7, 2025 · 2 comments

Comments

@Kyo1234567
Copy link

🔎 Search before asking

  • I have searched the PaddleOCR Docs and found no similar bug report.
  • I have searched the PaddleOCR Issues and found no similar bug report.
  • I have searched the PaddleOCR Discussions and found no similar bug report.

🐛 Bug (问题描述)

我在fine-tune 模型 ch_PP-OCRv4_det_server时,训练能正常进行,但是到eval时报GPU out of memory。我是用paddleocr 2.7的源代码训练的,batch_size=1, 我用Tesla T4 (16GB)的显卡eval都第1张时报oom, 我用A10 (24GB)显卡时 eval到大概第7张报oom。我尝试将paddlepaddle-gpu升级到3.0.0b1,问题依旧。我也试过AI Studio 上的pipe-line上传样本训练,是可以eval通过,但是需要充值。请问大佬们,有没有办法解决这个问题?

🏃‍♂️ Environment (运行环境)

OS: windows 10
paddleocr: 2.7 源码
paddlepaddle-gpu: 3.0.0b1
cuda: 11.8
cudnn: 8.9.7.29

🌰 Minimal Reproducible Example (最小可复现问题的Demo)

标准功能,不需要提供

@Topdu
Copy link
Collaborator

Topdu commented Feb 13, 2025

这可能是测试集中存在大分辨率的图片,导致推理时显存增加。你可以尝试在推理时,限制图片的最长边。

@Kyo1234567
Copy link
Author

谢谢你的回复。我使用ch_PP-OCRv4_det_server的推理模型直接推理是没有问题的。问题就出在fine-tune这个预训练模型的eval阶段(OOM)。
我想进一步请教一下,我知道在推理的时候可以使用参数“det_limit_side_len”和“det_limit_type”来控制输入图像的最长边,但是我不知道在训练的Eval阶段如何控制图像的最长边,能再指导一下吗?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants