Influence functions estimate effect of individual data points on predictions
of the model on test data and were adapted to deep learning in Koh and Liang
[2017]. They have been used for detecting data poisoning, detecting helpful and
harmful examples, influence of groups of datapoints, etc. Recently, Ilyas et
al. [2022] introduced a linear regression method they termed datamodels to
predict the effect of training points on outputs on test data. The current
paper seeks to provide a better theoretical understanding of such interesting
empirical phenomena. The primary tool is harmonic analysis and the idea of
noise stability. Contributions include: (a) Exact characterization of the
learnt datamodel in terms of Fourier coefficients. (b) An efficient method to
estimate the residual error and quality of the optimum linear datamodel without
having to train the datamodel. (c) New insights into when influences of groups
of datapoints may or may not add up linearly.

本文介绍了数据点对于模型预测的影响函数以及一种线性回归方法 datamodels 来预测训练点对测试数据输出的影响，并提供了更好的理论理解，主要工具是谐波分析和噪声稳定性。

通过谐波分析理解影响函数和数据模型

Understanding Influence Functions and Datamodels via Harmonic Analysis

A widely used algorithm for transfer learning is fine-tuning, where a
pre-trained model is fine-tuned on a target task with a small amount of labeled
data. When the capacity of the pre-trained model is much larger than the size
of the target data set, fine-tuning is prone to overfitting and "memorizing"
the training labels. Hence, an important question is to regularize fine-tuning
and ensure its robustness to noise. To address this question, we begin by
analyzing the generalization properties of fine-tuning. We present a PAC-Bayes
generalization bound that depends on the distance traveled in each layer during
fine-tuning and the noise stability of the fine-tuned model. We empirically
measure these quantities. Based on the analysis, we propose regularized
self-labeling -- the interpolation between regularization and self-labeling
methods, including (i) layer-wise regularization to constrain the distance
traveled in each layer; (ii) self label-correction and label-reweighting to
correct mislabeled data points (that the model is confident) and reweight less
confident data points. We validate our approach on an extensive collection of
image and text data sets using multiple pre-trained model architectures. Our
approach improves baseline methods by 1.76% (on average) for seven image
classification tasks and 0.75% for a few-shot classification task. When the
target data set includes noisy labels, our approach outperforms baseline
methods by 3.56% on average in two noisy settings.

通过 fine-tuning 算法的正则化与 self-labeling 方法相互插值，从而提高在小数据集下的图像与文本分类表现。

神经网络微调的改进正则化和鲁棒性

Improved Regularization and Robustness for Fine-tuning in Neural  Networks

Residual Neural Networks (ResNets) achieve state-of-the-art performance in
many computer vision problems. Compared to plain networks without residual
connections (PlnNets), ResNets train faster, generalize better, and suffer less
from the so-called degradation problem. We introduce simplified (but still
nonlinear) versions of ResNets and PlnNets for which these discrepancies still
hold, although to a lesser degree. We establish a 1-1 mapping between
simplified ResNets and simplified PlnNets, and show that they are exactly
equivalent to each other in expressive power for the same computational
complexity. We conjecture that ResNets generalize better because they have
better noise stability, and empirically support it for both simplified and
fully-fledged networks.

本研究比较 ResNets 和 PlnNets 的训练效率、泛化能力、容错率等性能，并探讨了两者之间简化版本的等价性，进而猜测 ResNets 泛化能力更强的原因与其较高的噪音容错能力有关，并通过实验支持了该猜想。

残差网络中的身份连接提高噪声稳定性

Identity Connections in Residual Nets Improve Noise Stability

Deep nets generalize well despite having more parameters than the number of
training samples. Recent works try to give an explanation using PAC-Bayes and
Margin-based analyses, but do not as yet result in sample complexity bounds
better than naive parameter counting. The current paper shows generalization
bounds that're orders of magnitude better in practice. These rely upon new
succinct reparametrizations of the trained net --- a compression that is
explicit and efficient. These yield generalization bounds via a simple
compression-based framework introduced here. Our results also provide some
theoretical justification for widespread empirical success in compressing deep
nets. Analysis of correctness of our compression relies upon some newly
identified \textquotedblleft noise stability\textquotedblright properties of
trained deep nets, which are also experimentally verified. The study of these
properties and resulting generalization bounds are also extended to
convolutional nets, which had eluded earlier attempts on proving
generalization.

本论文通过新的简洁再参数化方法的引入和新的 “噪声稳定性” 属性的发现，提供了一种基于压缩的框架来估计神经网络的泛化性能边界，并将此方法拓展到卷积神经网络中。