Energy-based models (EBMs) exhibit a variety of desirable properties in
predictive tasks, such as generality, simplicity and compositionality. However,
training EBMs on high-dimensional datasets remains unstable and expensive. In
this paper, we present a Manifold EBM (M-EBM) to boost the overall performance
of unconditional EBM and Joint Energy-based Model (JEM). Despite its
simplicity, M-EBM significantly improves unconditional EBMs in training
stability and speed on a host of benchmark datasets, such as CIFAR10, CIFAR100,
CelebA-HQ, and ImageNet 32x32. Once class labels are available,
label-incorporated M-EBM (M-JEM) further surpasses M-EBM in image generation
quality with an over 40% FID improvement, while enjoying improved accuracy. The
code can be found at this https URL

本研究提出了一种流形能量模型（M-EBM），以提高 unconditional EBM 和 Joint Energy-based Model (JEM) 的整体性能，在众多基准数据集上均显着提高训练稳定性和速度，在有类标签的情况下，标签集成 M-EBM (M-JEM) 进一步提高了图像生成质量，FID 改善了 40% 以上，并且精度也有所提高。

M-EBM: 了解基于能量模型的流形

M-EBM: Towards Understanding the Manifolds of Energy-Based Models

Can we train a hybrid discriminative-generative model within a single
network? This question has recently been answered in the affirmative,
introducing the field of Joint Energy-based Model (JEM), which achieves high
classification accuracy and image generation quality simultaneously. Despite
recent advances, there remain two performance gaps: the accuracy gap to the
standard softmax classifier, and the generation quality gap to state-of-the-art
generative models. In this paper, we introduce a variety of training techniques
to bridge the accuracy gap and the generation quality gap of JEM. 1) We
incorporate a recently proposed sharpness-aware minimization (SAM) framework to
train JEM, which promotes the energy landscape smoothness and the
generalizability of JEM. 2) We exclude data augmentation from the maximum
likelihood estimate pipeline of JEM, and mitigate the negative impact of data
augmentation to image generation quality. Extensive experiments on multiple
datasets demonstrate that our SADA-JEM achieves state-of-the-art performances
and outperforms JEM in image classification, image generation, calibration,
out-of-distribution detection and adversarial robustness by a notable margin.
Our code is available at this https URL

本文介绍了一系列训练技术来填补 JEM 的准确性和生成质量之间的差距，包括使用 SAM 框架来促进 JEM 能量景观的平滑性和泛化能力，将数据增强从 JEM 的最大似然估计管道中排除，并缓解数据增强对图像生成质量的负面影响。大量实验证明，我们的 SADA-JEM 在图像分类、图像生成、校准、外部分布检测和对抗鲁棒性方面均取得了突出表现，优于 JEM。

缩小联合能量模型的性能差距

Towards Bridging the Performance Gaps of Joint Energy-based Models

Joint Energy-based Model (JEM) is a recently proposed hybrid model that
retains strong discriminative power of modern CNN classifiers, while generating
samples rivaling the quality of GAN-based approaches. In this paper, we propose
a variety of new training procedures and architecture features to improve JEM's
accuracy, training stability, and speed altogether. 1) We propose a proximal
SGLD to generate samples in the proximity of samples from the previous step,
which improves the stability. 2) We further treat the approximate maximum
likelihood learning of EBM as a multi-step differential game, and extend the
YOPO framework to cut out redundant calculations during backpropagation, which
accelerates the training substantially. 3) Rather than initializing SGLD chain
from random noise, we introduce a new informative initialization that samples
from a distribution estimated from training data. 4) This informative
initialization allows us to enable batch normalization in JEM, which further
releases the power of modern CNN architectures for hybrid modeling. Code:
this https URL

本篇论文针对 Joint Energy-based Model (JEM) 的稳定性、准确性和速度提出了多种新的训练流程和架构特征，其中包括使用 proximal SGLD 生成与之前步骤相邻的样本、将 EBM 的近似最大似然学习视为多步微分博弈，并扩展 YOPO 框架以削减反向传播过程中的冗余计算、采用从训练数据中估计的分布采样的新的信息性初始化，以及在 JEM 中应用批量归一化等。