Energy Discrepancy

This repo contains PyTorch implementation of the paper "Energy Discrepancies: A Score-Independent Loss for Energy-Based Models"

by Tobias Schröder, Zijing Ou, Jen Ning Lim, Yingzhen Li, Sebastian Vollmer, and Andrew Duncan.

We propose a novel loss function called Energy Discrepancy (ED) which does not rely on the computation of scores or expensive Markov chain Monte Carlo. We show that ED approaches the explicit score matching and negative log-likelihood loss under different limits, effectively interpolating between both. Consequently, minimum ED estimation overcomes the problem of nearsightedness encountered in score-based estimation methods, while also enjoying theoretical guarantees. Through numerical experiments, we demonstrate that ED learns low-dimensional data distributions faster and more accurately than explicit score matching or contrastive divergence. For high-dimensional image data, we describe how the manifold hypothesis puts limitations on our approach and demonstrate the effectiveness of energy discrepancy by training the energy-based model as a prior of a variational decoder model.

Experiments

We provide the notebooks to reproduce the experiments in the paper:

• myopia_ed_sm_mle.ipynb: Reproduce Figure 1 (comparison of nearsightedness of Energy Discrepancy, Maximum Likelihood Estimation, and Score Matching)
• w_stabilisation.ipynb: Reproduce Figure 2 (understanding the w-stabilisation)
• density_estimation.ipynb: Density estimation on 2D toy data
• edlebm_mnist.ipynb: Training ED-LEBM on MNIST

Please also check this repository for the code of discrete energy discrepancy!!!