In their thought-provoking paper [1], Belkin et al. illustrate and discuss
the shape of risk curves in the context of modern high-complexity learners.
Given a fixed training sample size $n$, such curves show the risk of a learner
as a function of some (approximate) measure of its complexity $N$. With $N$ the
number of features, these curves are also referred to as feature curves. A
salient observation in [1] is that these curves can display, what they call,
double descent: with increasing $N$, the risk initially decreases, attains a
minimum, and then increases until $N$ equals $n$, where the training data is
fitted perfectly. Increasing $N$ even further, the risk decreases a second and
final time, creating a peak at $N=n$. This twofold descent may come as a
surprise, but as opposed to what [1] reports, it has not been overlooked
historically. Our letter draws attention to some original, earlier findings, of
interest to contemporary machine learning.

本文探讨现代高复杂度学习者的风险曲线，讨论风险曲线与学习器复杂度的关系，特别是所谓的双重下降现象，这一现象历史上早已被了解并已应用于当今的机器学习领域。

双重衰退的简要史

A Brief Prehistory of Double Descent

Traditionally in regression one minimizes the number of fitting parameters or
uses smoothing/regularization to trade training (TE) and generalization error
(GE). Driving TE to zero by increasing fitting degrees of freedom (dof) is
expected to increase GE. However modern big-data approaches, including deep
nets, seem to over-parametrize and send TE to zero (data interpolation) without
impacting GE. Overparametrization has the benefit that global minima of the
empirical loss function proliferate and become easier to find. These phenomena
have drawn theoretical attention. Regression and classification algorithms have
been shown that interpolate data but also generalize optimally. An interesting
related phenomenon has been noted: the existence of non-monotonic risk curves,
with a peak in GE with increasing dof. It was suggested that this peak
separates a classical regime from a modern regime where over-parametrization
improves performance. Similar over-fitting peaks were reported previously
(statistical physics approach to learning) and attributed to increased fitting
model flexibility. We introduce a generative and fitting model pair
("Misparametrized Sparse Regression" or MiSpaR) and show that the overfitting
peak can be dissociated from the point at which the fitting function gains
enough dof's to match the data generative model and thus provides good
generalization. This complicates the interpretation of overfitting peaks as
separating a "classical" from a "modern" regime. Data interpolation itself
cannot guarantee good generalization: we need to study the interpolation with
different penalty terms. We present analytical formulae for GE curves for
MiSpaR with $l_2$ and $l_1$ penalties, in the interpolating limit
$\lambda\rightarrow 0$.These risk curves exhibit important differences and help
elucidate the underlying phenomena.

本研究通过引入 MiSpaR（Misparametrized Sparse Regression）模型，表明过拟合峰值不能将经典阶段从现代阶段分隔开，数据内插本身不能保证良好的泛化能力，需要研究不同惩罚项下的内插状态。