Normal that real time for training increases ?

[Matcom274]

Literally, is it normal that the training time per epoch (real time or wall time) increases as training evolves just like the attached image.

I have tested with 5 models with adjusting several parameter values such as l_max, rcutoff, even though these parameters migh not be related... The drop of the line indicates when I restart the training using append=true setting.
It seems the usage of gpu memory slightly increased after 20 epoch but saturated at last. Would it be the reason? I thought that the model complexity is initially fixed (fixed memory) and the weights are the only trainable values. Maybe I don't fully understand how GNN exactly works.

• wall time and epoch from 'metrics_epoch.csv' log file.

settings

• almost the same with example.yaml
• training data is my own around ~5000frames with 70 atoms per cell.
• cuda11.3 / torch1.11.0 / A6000 or rtx3090

Thank you for your help in advance..!
Sorry if this question has been already discussed. I tried to search but couldn't find it.

[Matcom274]

Oh.. I just noticed it is accumulated time.

[Linux-cpp-lisp]

Hi @Matcom274 ,

Correct, it is cumulative time; this looks normal.

That said, there has previously been reported #311, so if you ever observe that issue, please do post in that discussion.