High-quality samples generated with score-based reverse diffusion algorithms
provide evidence that deep neural networks (DNN) trained for denoising can
learn high-dimensional densities, despite the curse of dimensionality. However,
recent reports of memorization of the training set raise the question of
whether these networks are learning the "true" continuous density of the data.
Here, we show that two denoising DNNs trained on non-overlapping subsets of a
dataset learn nearly the same score function, and thus the same density, with a
surprisingly small number of training images. This strong generalization
demonstrates an alignment of powerful inductive biases in the DNN architecture
and/or training algorithm with properties of the data distribution. We analyze
these, demonstrating that the denoiser performs a shrinkage operation in a
basis adapted to the underlying image. Examination of these bases reveals
oscillating harmonic structures along contours and in homogeneous image
regions. We show that trained denoisers are inductively biased towards these
geometry-adaptive harmonic representations by demonstrating that they arise
even when the network is trained on image classes such as low-dimensional
manifolds, for which the harmonic basis is suboptimal. Additionally, we show
that the denoising performance of the networks is near-optimal when trained on
regular image classes for which the optimal basis is known to be
geometry-adaptive and harmonic.

通过基于分数的逆扩散算法生成的高质量样本提供了证据，表明尽管遭受维度灾难的困扰，用于降噪训练的深度神经网络（DNN）可以学习高维密度。然而，关于训练集记忆化的最近报导引发了一个问题，即这些网络是否正在学习数据的 “真实” 连续密度。在本文中，我们展示了在非重叠的数据集子集上训练的两个降噪 DNN 学习到几乎相同的评分函数，从而学习到相同的密度，并且只需要非常少的训练图像。这种强大的泛化表明 DNN 架构和 / 或训练算法中的强大归纳偏差与数据分布的特性相一致。我们通过对这些内容进行分析来证明这一点，证明了去噪器在适应底层图像的基础上执行了一个收缩操作。对这些基础的检查揭示了轮廓线和均匀图像区域中的振荡谐波结构。我们通过证明即使在训练于低维流形等图像类别的情况下，这些经过训练的去噪器也会生成这种几何自适应谐波表示来表明它们在归纳偏差方面具有偏好。此外，我们还展示了当在已知最优基础为几何自适应谐波的常规图像类别上进行训练时，网络的去噪性能接近最优。