How to continue a previously terminated training process?

[kuntalg97]

Hi,

Is there a way to continue a previously terminated training process? I'm trying to train an Nequip model using nequip-train and my run got terminated due to some cluster issues before any of the stopping criteria were met. My results directory has been generated with the log file as well. How do I continue this run, from the point where the training stopped?

I've also posted this question in the Allegro discussions forum, but I assume the answer would be the same.

Thanks!
Kuntal

[terryfrankcombe]

If you restart the job it should pick up from where you left off. Note the append key in the yaml.

[kuntalg97]

Thanks! Yes I set the append to be True in the yaml file. It worked.

[Linux-cpp-lisp]

See also mir-group/allegro#96