Deep learning is renowned for its theory-practice gap, whereby principled
theory typically fails to provide much beneficial guidance for implementation
in practice. This has been highlighted recently by the benign overfitting
phenomenon: when neural networks become sufficiently large to interpolate the
dataset perfectly, model performance appears to improve with increasing model
size, in apparent contradiction with the well-known bias-variance tradeoff.
While such phenomena have proven challenging to theoretically study for general
models, the recently proposed Interpolating Information Criterion (IIC)
provides a valuable theoretical framework to examine performance for
overparameterized models. Using the IIC, a PAC-Bayes bound is obtained for a
general class of models, characterizing factors which influence generalization
performance in the interpolating regime. From the provided bound, we quantify
how the test error for overparameterized models achieving effectively zero
training error depends on the quality of the implicit regularization imposed by
e.g. the combination of model, optimizer, and parameter-initialization scheme;
the spectrum of the empirical neural tangent kernel; curvature of the loss
landscape; and noise present in the data.

深度学习在理论和实践之间存在理论实践差距，最近的研究关注了过拟合现象及其影响因素，提出了插值信息准则来研究过参数化模型的性能和泛化能力。

关于插值信息准则的 PAC-Bayesian 观点

A PAC-Bayesian Perspective on the Interpolating Information Criterion

The problem of model selection is considered for the setting of interpolating
estimators, where the number of model parameters exceeds the size of the
dataset. Classical information criteria typically consider the large-data
limit, penalizing model size. However, these criteria are not appropriate in
modern settings where overparameterized models tend to perform well. For any
overparameterized model, we show that there exists a dual underparameterized
model that possesses the same marginal likelihood, thus establishing a form of
Bayesian duality. This enables more classical methods to be used in the
overparameterized setting, revealing the Interpolating Information Criterion, a
measure of model quality that naturally incorporates the choice of prior into
the model selection. Our new information criterion accounts for prior
misspecification, geometric and spectral properties of the model, and is
numerically consistent with known empirical and theoretical behavior in this
regime.

本研究考虑了拟合估计器模型选择问题，其中模型参数的数量超过数据集的大小。研究者们通过建立双重欠定参数模型来解决了模型选择问题，提出了一种新的统计量，称为插值信息准则（Interpolating Information Criterion）。这个统计量不仅可以衡量模型拟合的质量，而且还可以自然地纳入先验选择问题。