Overparameterized neural networks (NNs) are observed to generalize well even
when trained to perfectly fit noisy data. This phenomenon motivated a large
body of work on "benign overfitting", where interpolating predictors achieve
near-optimal performance. Recently, it was conjectured and empirically observed
that the behavior of NNs is often better described as "tempered overfitting",
where the performance is non-optimal yet also non-trivial, and degrades as a
function of the noise level. However, a theoretical justification of this claim
for non-linear NNs has been lacking so far. In this work, we provide several
results that aim at bridging these complementing views. We study a simple
classification setting with 2-layer ReLU NNs, and prove that under various
assumptions, the type of overfitting transitions from tempered in the extreme
case of one-dimensional data, to benign in high dimensions. Thus, we show that
the input dimension has a crucial role on the type of overfitting in this
setting, which we also validate empirically for intermediate dimensions.
Overall, our results shed light on the intricate connections between the
dimension, sample size, architecture and training algorithm on the one hand,
and the type of resulting overfitting on the other hand.

本文研究过参数神经网络的过拟合现象，并证明了在不同的维度下会出现不同种类的过拟合现象，包括良性过拟合和温和过拟合，并解释了这些现象与样本大小、网络结构等因素的相关性。

从 tempered 到 benign 的 ReLU 神经网络过拟合

From Tempered to Benign Overfitting in ReLU Neural Networks

The practical success of overparameterized neural networks has motivated the
recent scientific study of interpolating methods, which perfectly fit their
training data. Certain interpolating methods, including neural networks, can
fit noisy training data without catastrophically bad test performance, in
defiance of standard intuitions from statistical learning theory. Aiming to
explain this, a body of recent work has studied benign overfitting, a
phenomenon where some interpolating methods approach Bayes optimality, even in
the presence of noise. In this work we argue that while benign overfitting has
been instructive and fruitful to study, many real interpolating methods like
neural networks do not fit benignly: modest noise in the training set causes
nonzero (but non-infinite) excess risk at test time, implying these models are
neither benign nor catastrophic but rather fall in an intermediate regime. We
call this intermediate regime tempered overfitting, and we initiate its
systematic study. We first explore this phenomenon in the context of kernel
(ridge) regression (KR) by obtaining conditions on the ridge parameter and
kernel eigenspectrum under which KR exhibits each of the three behaviors. We
find that kernels with powerlaw spectra, including Laplace kernels and ReLU
neural tangent kernels, exhibit tempered overfitting. We then empirically study
deep neural networks through the lens of our taxonomy, and find that those
trained to interpolation are tempered, while those stopped early are benign. We
hope our work leads to a more refined understanding of overfitting in modern
learning.

该论文研究了神经网络等插值方法是否能够在存在噪声的情况下，拟合训练数据而不会表现出灾难性的测试性能，尝试通过 “良性过拟合” 和 “温和过拟合” 两个现象进行解释，并首次系统研究了 “温和过拟合” 的性质及在核（岭）回归中的表现，以及在深度神经网络中的实验结果。