[Feature Request] L1 L2 Regularization

[well-zt]

Summary

Hi, I'm using Deepmd-kit, but I encountered over-fitting problem during training. How can I port L1 L2 regularization into deepmd-kit?

Detailed Description

Just as summary

Further Information, Files, and Links

No response

[caic99]

Hi @well-zt ,
You can try setting gradient_max_norm parameter in input.json. The related codes are here:

deepmd-kit/deepmd/pt/train/training.py

Lines 689 to 694 in d162d0b

	if self.gradient_max_norm > 0.0:
	torch.nn.utils.clip_grad_norm_(
	self.wrapper.parameters(),
	self.gradient_max_norm,
	error_if_nonfinite=True,
	)

Ref

[QuantumMisaka]

@caic99 L1, L2 regularization is a common practice in ML training but not common in MLIP training, should we put how (and why) to use regularization in loss function in docs of deepmd-kit ?

[caic99]

@caic99 L1, L2 regularization is a common practice in ML training but not common in MLIP training, should we put how (and why) to use regularization in loss function in docs of deepmd-kit ?

@QuantumMisaka I would like to correct my reply above: what I meant was the L2 norm used for gradient clipping, which is not the same thing as the original poster said.

how (and why) to use regularization in loss function

Unluckily I have no experience on using regularization for MLIP.