Modern deep learning models employ considerably more parameters than required
to fit the training data. Whereas conventional statistical wisdom suggests such
models should drastically overfit, in practice these models generalize
remarkably well. An emerging paradigm for describing this unexpected behavior
is in terms of a \emph{double descent} curve, in which increasing a model's
capacity causes its test error to first decrease, then increase to a maximum
near the interpolation threshold, and then decrease again in the
overparameterized regime. Recent efforts to explain this phenomenon
theoretically have focused on simple settings, such as linear regression or
kernel regression with unstructured random features, which we argue are too
coarse to reveal important nuances of actual neural networks. We provide a
precise high-dimensional asymptotic analysis of generalization under kernel
regression with the Neural Tangent Kernel, which characterizes the behavior of
wide neural networks optimized with gradient descent. Our results reveal that
the test error has non-monotonic behavior deep in the overparameterized regime
and can even exhibit additional peaks and descents when the number of
parameters scales quadratically with the dataset size.

深度学习模型参数通常大于所需，然而其测试误差在过拟合阈值附近有极值和下降，在过参数化区间反而下降，神经切向核模型可以提供有关真实神经网络的细节。

高维神经切向核：三倍下降与广义化的多尺度理论

The Neural Tangent Kernel in High Dimensions: Triple Descent and a  Multi-Scale Theory of Generalization

This paper explores the generalization loss of linear regression in variably
parameterized families of models, both under-parameterized and
over-parameterized. We show that the generalization curve can have an arbitrary
number of peaks, and moreover, locations of those peaks can be explicitly
controlled. Our results highlight the fact that both classical U-shaped
generalization curve and the recently observed double descent curve are not
intrinsic properties of the model family. Instead, their emergence is due to
the interaction between the properties of the data and the inductive biases of
learning algorithms.

该研究探讨了可变参数模型家族中线性回归的泛化损失，证明了一般化曲线可以有任意数量的峰值，并且这些峰值位置可以明确地受到控制。结果表明，经典的 U 形一般化曲线和最近观察到的双下降曲线不是模型家族的固有属性，而是由数据和学习算法的归纳偏差相互作用所导致的。

多重下降：设计自己的泛化曲线

Multiple Descent: Design Your Own Generalization Curve

Breakthroughs in machine learning are rapidly changing science and society,
yet our fundamental understanding of this technology has lagged far behind.
Indeed, one of the central tenets of the field, the bias-variance trade-off,
appears to be at odds with the observed behavior of methods used in the modern
machine learning practice. The bias-variance trade-off implies that a model
should balance under-fitting and over-fitting: rich enough to express
underlying structure in data, simple enough to avoid fitting spurious patterns.
However, in the modern practice, very rich models such as neural networks are
trained to exactly fit (i.e., interpolate) the data. Classically, such models
would be considered over-fit, and yet they often obtain high accuracy on test
data. This apparent contradiction has raised questions about the mathematical
foundations of machine learning and their relevance to practitioners.
In this paper, we reconcile the classical understanding and the modern
practice within a unified performance curve. This "double descent" curve
subsumes the textbook U-shaped bias-variance trade-off curve by showing how
increasing model capacity beyond the point of interpolation results in improved
performance. We provide evidence for the existence and ubiquity of double
descent for a wide spectrum of models and datasets, and we posit a mechanism
for its emergence. This connection between the performance and the structure of
machine learning models delineates the limits of classical analyses, and has
implications for both the theory and practice of machine learning.

本文中，我们通过统一的性能曲线，协调了传统理解与现代实践，它包含了传统的 U 形偏差 - 方差权衡曲线，这个被称为 “双下降” 曲线的统计证据，证明了其存在于各种模型和数据集中，并推断了其出现机制。通过机器学习模型性能与结构之间的联系，勾勒出了传统分析的局限性，对机器学习的理论和实践都有重要意义。