Double-descent refers to the unexpected drop in test loss of a learning
algorithm beyond an interpolating threshold with over-parameterization, which
is not predicted by information criteria in their classical forms due to the
limitations in the standard asymptotic approach. We update these analyses using
the information risk minimization framework and provide Akaike Information
Criterion (AIC) and Bayesian Information Criterion (BIC) for models learned by
stochastic gradient Langevin dynamics (SGLD). Notably, the AIC and BIC penalty
terms for SGLD correspond to specific information measures, i.e., symmetrized
KL information and KL divergence. We extend this information-theoretic analysis
to over-parameterized models by characterizing the SGLD-based BIC for the
random feature model in the regime where the number of parameters $p$ and the
number of samples $n$ tend to infinity, with $p/n$ fixed. Our experiments
demonstrate that the refined SGLD-based BIC can track the double-descent curve,
providing meaningful guidance for model selection and revealing new insights
into the behavior of SGLD learning algorithms in the over-parameterized regime.

通过信息风险最小化框架更新信息准则定理的分析，提供了基于随机梯度 langevin 动力学的模型的赤池信息准则和贝叶斯信息准则，并将信息论分析扩展到了过参数化模型中。

基于 SGLD 的信息准则和过参数化区域

SGLD-Based Information Criteria and the Over-Parameterized Regime

In this expository note we describe a surprising phenomenon in
overparameterized linear regression, where the dimension exceeds the number of
samples: there is a regime where the test risk of the estimator found by
gradient descent increases with additional samples. In other words, more data
actually hurts the estimator. This behavior is implicit in a recent line of
theoretical works analyzing "double-descent" phenomenon in linear models. In
this note, we isolate and understand this behavior in an extremely simple
setting: linear regression with isotropic Gaussian covariates. In particular,
this occurs due to an unconventional type of bias-variance tradeoff in the
overparameterized regime: the bias decreases with more samples, but variance
increases.

在高参数化线性回归领域中，出现了一个令人惊讶的现象，即当样本数少于特征维数时，测试误差会随着样本增加而增加，即更多的数据反而会伤害估计器。该现象是近期理论研究分析线性模型中 “双下降” 现象的一部分。在本文中，我们以线性回归为预测模型，通过对具有各向同性高斯协方差矩阵的数据集进行分析，对该现象进行了总结和阐述。