Score-based generative models have excellent performance in terms of
generation quality and likelihood. They model the data distribution by matching
a parameterized score network with first-order data score functions. The score
network can be used to define an ODE ("score-based diffusion ODE") for exact
likelihood evaluation. However, the relationship between the likelihood of the
ODE and the score matching objective is unclear. In this work, we prove that
matching the first-order score is not sufficient to maximize the likelihood of
the ODE, by showing a gap between the maximum likelihood and score matching
objectives. To fill up this gap, we show that the negative likelihood of the
ODE can be bounded by controlling the first, second, and third-order score
matching errors; and we further present a novel high-order denoising score
matching method to enable maximum likelihood training of score-based diffusion
ODEs. Our algorithm guarantees that the higher-order matching error is bounded
by the training error and the lower-order errors. We empirically observe that
by high-order score matching, score-based diffusion ODEs achieve better
likelihood on both synthetic data and CIFAR-10, while retaining the high
generation quality.

该研究论文介绍了如何通过高阶噪声抑制分数匹配方法实现得分网络的最大似然训练，以提高得分模型的生成质量和对于数据概率分布的似然评估。

高阶降噪得分匹配用于分数型扩散 ODE 的最大似然训练

Maximum Likelihood Training for Score-Based Diffusion ODEs by High-Order  Denoising Score Matching

Denoising diffusion probabilistic models (DDPMs) have emerged as competitive
generative models yet brought challenges to efficient sampling. In this paper,
we propose novel bilateral denoising diffusion models (BDDMs), which take
significantly fewer steps to generate high-quality samples. From a bilateral
modeling objective, BDDMs parameterize the forward and reverse processes with a
score network and a scheduling network, respectively. We show that a new lower
bound tighter than the standard evidence lower bound can be derived as a
surrogate objective for training the two networks. In particular, BDDMs are
efficient, simple-to-train, and capable of further improving any pre-trained
DDPM by optimizing the inference noise schedules. Our experiments demonstrated
that BDDMs can generate high-fidelity samples with as few as 3 sampling steps
and produce comparable or even higher quality samples than DDPMs using 1000
steps with only 16 sampling steps (a 62x speedup).

该论文提出了新的双边去噪扩散模型（BDDMs）来提高样本生成的质量和速度，其可以更加高效和简单地训练，以及进一步优化先前训练好的模型。

双边去噪扩散模型

Bilateral Denoising Diffusion Models

Recent advances in diffusion models bring state-of-the-art performance on
image generation tasks. However, empirical results from previous research in
diffusion models imply an inverse correlation between density estimation and
sample generation performances. This paper investigates with sufficient
empirical evidence that such inverse correlation happens because density
estimation is significantly contributed by small diffusion time, whereas sample
generation mainly depends on large diffusion time. However, training a score
network well across the entire diffusion time is demanding because the loss
scale is significantly imbalanced at each diffusion time. For successful
training, therefore, we introduce Soft Truncation, a universally applicable
training technique for diffusion models, that softens the fixed and static
truncation hyperparameter into a random variable. In experiments, Soft
Truncation achieves state-of-the-art performance on CIFAR-10, CelebA, CelebA-HQ
256x256, and STL-10 datasets.

研究表明，扩散模型中样本生成的表现与密度估计呈负相关，这是因为密度估计对小扩散时间的贡献较大，而样本生成主要依赖于大扩散时间；为了成功训练模型，我们引入了一种通用的技术 —— 软截断，该技术将固定的截断超参数转化为随机变量，实验证明该方法在数据集 CIFAR-10，CelebA，CelebA-HQ 256x256 和 STL-10 上取得了最新的最佳表现。