The phenomenon of benign overfitting is one of the key mysteries uncovered by
deep learning methodology: deep neural networks seem to predict well, even with
a perfect fit to noisy training data. Motivated by this phenomenon, we consider
when a perfect fit to training data in linear regression is compatible with
accurate prediction. We give a characterization of linear regression problems
for which the minimum norm interpolating prediction rule has near-optimal
prediction accuracy. The characterization is in terms of two notions of the
effective rank of the data covariance. It shows that overparameterization is
essential for benign overfitting in this setting: the number of directions in
parameter space that are unimportant for prediction must significantly exceed
the sample size. By studying examples of data covariance properties that this
characterization shows are required for benign overfitting, we find an
important role for finite-dimensional data: the accuracy of the minimum norm
interpolating prediction rule approaches the best possible accuracy for a much
narrower range of properties of the data distribution when the data lies in an
infinite dimensional space versus when the data lies in a finite dimensional
space whose dimension grows faster than the sample size.

研究表明，过度拟合是深度学习方法的主要问题之一，但当训练中使用最小规范化规则时，线性回归问题中的过度拟合也可以实现高精度预测，需要超参数数量显著超过样本大小。