An energy-based model (EBM) is a popular generative framework that offers
both explicit density and architectural flexibility, but training them is
difficult since it is often unstable and time-consuming. In recent years,
various training techniques have been developed, e.g., better divergence
measures or stabilization in MCMC sampling, but there often exists a large gap
between EBMs and other generative frameworks like GANs in terms of generation
quality. In this paper, we propose a novel and effective framework for
improving EBMs via contrastive representation learning (CRL). To be specific,
we consider representations learned by contrastive methods as the true
underlying latent variable. This contrastive latent variable could guide EBMs
to understand the data structure better, so it can improve and accelerate EBM
training significantly. To enable the joint training of EBM and CRL, we also
design a new class of latent-variable EBMs for learning the joint density of
data and the contrastive latent variable. Our experimental results demonstrate
that our scheme achieves lower FID scores, compared to prior-art EBM methods
(e.g., additionally using variational autoencoders or diffusion techniques),
even with significantly faster and more memory-efficient training. We also show
conditional and compositional generation abilities of our latent-variable EBMs
as their additional benefits, even without explicit conditional training. The
code is available at this https URL

本文提出了一种基于对比学习的框架，通过引入对比潜变量来改进能量基模型的训练，设计一种新类的潜变量 EBMs 实现数据联合密度估计和对比潜变量的联合训练，实验结果表明该方案比现有的 EBM 方法（如变分自动编码器或扩散技术）具有更低的 FID 分数，训练速度更快、内存效率更高，并展示了该潜变量 EBMs 的条件和组合生成能力。