In this paper, we propose a novel Energy-Calibrated Generative Model that
utilizes a Conditional EBM for enhancing Variational Autoencoders (VAEs). VAEs
are sampling efficient but often suffer from blurry generation results due to
the lack of training in the generative direction. On the other hand,
Energy-Based Models (EBMs) can generate high-quality samples but require
expensive Markov Chain Monte Carlo (MCMC) sampling. To address these issues, we
introduce a Conditional EBM for calibrating the generative direction during
training, without requiring it for test time sampling. Our approach enables the
generative model to be trained upon data and calibrated samples with adaptive
weight, thereby enhancing efficiency and effectiveness without necessitating
MCMC sampling in the inference phase. We also show that the proposed approach
can be extended to calibrate normalizing flows and variational posterior.
Moreover, we propose to apply the proposed method to zero-shot image
restoration via neural transport prior and range-null theory. We demonstrate
the effectiveness of the proposed method through extensive experiments in
various applications, including image generation and zero-shot image
restoration. Our method shows state-of-the-art performance over single-step
non-adversarial generation.

本文提出了一种新颖的能量校准生成模型，该模型利用条件 EBM 来增强变分自动编码器（VAEs）的性能，针对 VAEs 在生成方向上训练不足而导致生成结果模糊的问题，以及 EBMs 需要昂贵的马尔可夫链蒙特卡罗（MCMC）采样的问题，我们引入了条件 EBM 用于训练时校准生成方向，而不需要在测试时进行采样，这样的方法使得生成模型能够根据数据和校准样本进行自适应加权的训练，从而提高了效率和效果，而不需要在推理阶段进行 MCMC 采样，我们还展示了该方法可以扩展到校准归一化流和变分后验，并提出将该方法应用于基于神经传输先验和零射图像恢复，通过广泛的实验在图像生成和零射图像恢复等各种应用中证明了该方法的有效性，我们的方法在单步非对抗式生成方面展现了最先进的性能。