Common practice in modern machine learning involves fitting a large number of parameters relative to the number of observations. These overparameterized models can exhibit surprising generalization behavior, e.g., ``double descent'' in the prediction error curve when plotted against the raw number of model parameters, or another simplistic notion of complexity. In this paper, we revisit model complexity from first principles, by first reinterpreting and then extending the classical statistical concept of (effective) degrees of freedom. Whereas the classical definition is connected to fixed-X prediction error (in which prediction error is defined by averaging over the same, nonrandom covariate points as those used during training), our extension of degrees of freedom is connected to random-X prediction error (in which prediction error is averaged over a new, random sample from the covariate distribution). The random-X setting more naturally embodies modern machine learning problems, where highly complex models, even those complex enough to interpolate the training data, can still lead to desirable generalization performance under appropriate conditions. We demonstrate the utility of our proposed complexity measures through a mix of conceptual arguments, theory, and experiments, and illustrate how they can be used to interpret and compare arbitrary prediction models.

本研究解决了现代机器学习领域模型复杂性理解不足的问题。通过重新解释并扩展经典的（有效）自由度概念，提出了一种新的随机自变量预测误差度量方法，从而更好地说明过参数化模型在复杂性与泛化能力之间的关系。研究表明，在适当条件下，复杂模型即使能插值训练数据，仍能实现良好的泛化表现。

重新审视过参数化机器学习中的乐观主义与模型复杂性