TF32 vs FP32

[tgmaxson]

I am trying to push accuracy with Allegro down to relatively low values of <1 meV/atom and <20 meV/Angstrom for energies and forces and am wondering if the usage of TF32 is problematic or if there may be needs to go to 64 bit precision. Have you performed any tests that I can reference or do you have any suggestions on how to easily test what the effect actually is? Since all models train a little different I was wondering if it is possible to see the effects of the different precisions without retraining a model.

I will also be running some of my own tests, but some insight could be helpful since it seems there has been some discussion of this on the development end. This is on A100 GPUs in particular.

[Linux-cpp-lisp]

Hi @tgmaxson ,

You may have issues, but it's hard to say and depends on many different factors. However I would strongly recommend that if you are seeing numerics issues you migrate to our develop branch in the nequip repo, which contains some major developments on that front.

At present we disable TF32 entirely by default to avoid numerical pitfalls, however I'd be curious to hear about any results of your experiments on whether you see problems with it.