CUDA out of memory in the middle of fine tuning

[hirunima]

First of all thank you for providing the trainig scripts. I am using four A4000 with 16G memory each. While trying to fine tune on COCO, CUDA produce an OutOfMemoryError after about 5 steps which can be expected due to total memory increment indicated in each step. Is there any reason for this memory increase during the training?

[tz3458172]

First of all thank you for providing the trainig scripts. I am using four A4000 with 16G memory each. While trying to fine tune on COCO, CUDA produce an OutOfMemoryError after about 5 steps which can be expected due to total memory increment indicated in each step. Is there any reason for this memory increase during the training?

Same question. How did you solve it