Energy-based models (EBMs) are generative models inspired by statistical
physics with a wide range of applications in unsupervised learning. Their
performance is best measured by the cross-entropy (CE) of the model
distribution relative to the data distribution. Using the CE as the objective
for training is however challenging because the computation of its gradient
with respect to the model parameters requires sampling the model distribution.
Here we show how results for nonequilibrium thermodynamics based on Jarzynski
equality together with tools from sequential Monte-Carlo sampling can be used
to perform this computation efficiently and avoid the uncontrolled
approximations made using the standard contrastive divergence algorithm.
Specifically, we introduce a modification of the unadjusted Langevin algorithm
(ULA) in which each walker acquires a weight that enables the estimation of the
gradient of the cross-entropy at any step during GD, thereby bypassing sampling
biases induced by slow mixing of ULA. We illustrate these results with
numerical experiments on Gaussian mixture distributions as well as the MNIST
dataset. We show that the proposed approach outperforms methods based on the
contrastive divergence algorithm in all the considered situations.

本文介绍了一种基于 Jarzynski 平等和序贯蒙特卡洛抽样工具的改进型未校正 Langevin 算法来更有效地计算交叉熵的梯度，避免了标准对比散度算法中存在的不可控逼近问题，在高斯混合分布和 MNIST 数据集上的实验结果均显示该方法优于对比散度算法。