Score-based generative models have excellent performance in terms of
generation quality and likelihood. They model the data distribution by matching
a parameterized score network with first-order data score functions. The score
network can be used to define an ODE ("score-based diffusion ODE") for exact
likelihood evaluation. However, the relationship between the likelihood of the
ODE and the score matching objective is unclear. In this work, we prove that
matching the first-order score is not sufficient to maximize the likelihood of
the ODE, by showing a gap between the maximum likelihood and score matching
objectives. To fill up this gap, we show that the negative likelihood of the
ODE can be bounded by controlling the first, second, and third-order score
matching errors; and we further present a novel high-order denoising score
matching method to enable maximum likelihood training of score-based diffusion
ODEs. Our algorithm guarantees that the higher-order matching error is bounded
by the training error and the lower-order errors. We empirically observe that
by high-order score matching, score-based diffusion ODEs achieve better
likelihood on both synthetic data and CIFAR-10, while retaining the high
generation quality.

该研究论文介绍了如何通过高阶噪声抑制分数匹配方法实现得分网络的最大似然训练，以提高得分模型的生成质量和对于数据概率分布的似然评估。

高阶降噪得分匹配用于分数型扩散 ODE 的最大似然训练

Maximum Likelihood Training for Score-Based Diffusion ODEs by High-Order  Denoising Score Matching

Normalizing flows are among the most popular paradigms in generative
modeling, especially for images, primarily because we can efficiently evaluate
the likelihood of a data point. This is desirable both for evaluating the fit
of a model, and for ease of training, as maximizing the likelihood can be done
by gradient descent. However, training normalizing flows comes with
difficulties as well: models which produce good samples typically need to be
extremely deep -- which comes with accompanying vanishing/exploding gradient
problems. A very related problem is that they are often poorly conditioned:
since they are parametrized as invertible maps from $\mathbb{R}^d \to
\mathbb{R}^d$, and typical training data like images intuitively is
lower-dimensional, the learned maps often have Jacobians that are close to
being singular.
In our paper, we tackle representational aspects around depth and
conditioning of normalizing flows: both for general invertible architectures,
and for a particular common architecture, affine couplings. We prove that
$\Theta(1)$ affine coupling layers suffice to exactly represent a permutation
or $1 \times 1$ convolution, as used in GLOW, showing that representationally
the choice of partition is not a bottleneck for depth. We also show that
shallow affine coupling networks are universal approximators in Wasserstein
distance if ill-conditioning is allowed, and experimentally investigate related
phenomena involving padding. Finally, we show a depth lower bound for general
flow architectures with few neurons per layer and bounded Lipschitz constant.

本文研究了对于对数几率模型的归一化流表示，主要关注模型深度和分区选择实现上的困难。结果表明，每个分区仅需 Theta (1) 个仿射耦合层就足以精确表示排列或者 1*1 的卷积，同时也被证明具有较好的普适性。同时我们也展示了对于少量神经元元素和有限 Lipschitz 常数的流构架的深度下限。

正则化流中深度和条件化的表征特征

Representational aspects of depth and conditioning in normalizing flows

This paper is the first work to propose a network to predict a structured
uncertainty distribution for a synthesized image. Previous approaches have been
mostly limited to predicting diagonal covariance matrices. Our novel model
learns to predict a full Gaussian covariance matrix for each reconstruction,
which permits efficient sampling and likelihood evaluation.
We demonstrate that our model can accurately reconstruct ground truth
correlated residual distributions for synthetic datasets and generate plausible
high frequency samples for real face images. We also illustrate the use of
these predicted covariances for structure preserving image denoising.

本篇论文首次提出了一种网络模型，其可预测合成图像的结构化不确定性分布。模型可学习预测每个重构的完整高斯协方差矩阵，从而实现有效的采样和似然评估，通过该方法，能够准确地重构合成数据集的地面真实相关残差分布，并为真实面部图像生成可信度高的概率密度函数分布。同时，展示了合成图像的结构保留降噪方法。