How do score-based generative models (SBMs) learn the data distribution
supported on a low-dimensional manifold? We investigate the score model of a
trained SBM through its linear approximations and subspaces spanned by local
feature vectors. During diffusion as the noise decreases, the local
dimensionality increases and becomes more varied between different sample
sequences. Importantly, we find that the learned vector field mixes samples by
a non-conservative field within the manifold, although it denoises with normal
projections as if there is an energy function in off-manifold directions. At
each noise level, the subspace spanned by the local features overlap with an
effective density function. These observations suggest that SBMs can flexibly
mix samples with the learned score field while carefully maintaining a
manifold-like structure of the data distribution.

通过线性逼近和本地特征向量生成的子空间，调查经过训练的评分模型 (linear approximations and subspaces spanned by local feature vectors)，通过低维流形上支持的数据分布支持的得分模型 (score-based generative models) 学习数据分布是如何学习的。

基于评分的生成模型学习具有约束混合的流形结构

Score-based generative models learn manifold-like structures with  constrained mixing

For certain infinitely-wide neural networks, the neural tangent kernel (NTK)
theory fully characterizes generalization, but for the networks used in
practice, the empirical NTK only provides a rough first-order approximation.
Still, a growing body of work keeps leveraging this approximation to
successfully analyze important deep learning phenomena and design algorithms
for new applications. In our work, we provide strong empirical evidence to
determine the practical validity of such approximation by conducting a
systematic comparison of the behavior of different neural networks and their
linear approximations on different tasks. We show that the linear
approximations can indeed rank the learning complexity of certain tasks for
neural networks, even when they achieve very different performances. However,
in contrast to what was previously reported, we discover that neural networks
do not always perform better than their kernel approximations, and reveal that
the performance gap heavily depends on architecture, dataset size and training
task. We discover that networks overfit to these tasks mostly due to the
evolution of their kernel during training, thus, revealing a new type of
implicit bias.

该论文通过对神经网络和它们的线性近似在不同任务上的行为进行系统比较，提供了强有力的实证证据来确定近似的实际有效性，发现网络并不总是表现优于其核近似，并揭示了性能差距很大程度上取决于网络架构、数据集大小和训练任务。此外，研究发现网络在训练过程中出现过拟合的原因是其核的演化。由此揭示了一种新的隐式偏差现象。

线性化神经网络在泛化方面的实际表现

What can linearized neural networks actually say about generalization?

Linear approximations to the decision boundary of a complex model have become
one of the most popular tools for interpreting predictions. In this paper, we
study such linear explanations produced either post-hoc by a few recent methods
or generated along with predictions with contextual explanation networks
(CENs). We focus on two questions: (i) whether linear explanations are always
consistent or can be misleading, and (ii) when integrated into the prediction
process, whether and how explanations affect the performance of the model. Our
analysis sheds more light on certain properties of explanations produced by
different methods and suggests that learning models that explain and predict
jointly is often advantageous.

本文研究了产生于复杂模型后期的线性解释或与上下文解释网络 (CENs) 一起产生的线性解释。研究主要聚焦于线性解释是否一直是始终如一的或容易引导，同时研究在将其整合到预测过程中时，解释是否会影响模型程序的表现。我们的分析揭示了不同方法产生的解释的某些属性，并建议共同预测和解释的学习模型经常具有优势。