Using a pretrained model to continue training on new data: possible? What pitfalls?

[vputz]

We're investigating using a pretrained Allegro/Nequip model to continue training with new training data and were wondering if such a thing were possible and if it is what hazards may be present.

A few things that stand out is that some aspects of the model are initialized from the initial dataset and changed in the config (avg_num_neighbors and avg_num_atoms come to mind). As well, the PerSpeciesRescale and RescaleEnergyEtc builders clearly use data obtained by statistical analysis of the initial training dataset, which makes sense given this from the paper:

We normalize the target energies by subtracting the mean potential energy over the training set and scale both the target energies and target force components by the root mean square of the force components over the training set. The predicted atomic energies ^ Ei are scaled and shifted by two learnable per-species parameters before summing them for the total predicted potential energy ^ E:

But what if the full training set is unknown at model creation? For example, suppose we had 1000 molecules with known energies and train a model, then load that model add 1000 new molecules to the training set for additional training? Aside from the "normal" caveats (forgetting, etc) how would this "initialization" be affected?

• Would it be okay if they had the same elements?
• or similar energies?
• Would maybe conformers (identical elements but slightly varying geometries) likely be ok but different-sized molecules be unacceptable?

Or would this be really impossible and is the only reasonable path forward training the new model from scratch with the new data as a full "initial training set" each time we add data?

[vputz]

(or if the per-species rescale is being done in "dataset units", would it be enough to change the global rescale and average numbers based on the statistics of the new data set so long as the same set of atomic types were used, or something similar?)