The problem of benign overfitting asks whether it is possible for a model to
perfectly fit noisy training data and still generalize well. We study benign
overfitting in two-layer leaky ReLU networks trained with the hinge loss on a
binary classification task. We consider input data which can be decomposed into
the sum of a common signal and a random noise component, which lie on subspaces
orthogonal to one another. We characterize conditions on the signal to noise
ratio (SNR) of the model parameters giving rise to benign versus non-benign, or
harmful, overfitting: in particular, if the SNR is high then benign overfitting
occurs, conversely if the SNR is low then harmful overfitting occurs. We
attribute both benign and non-benign overfitting to an approximate margin
maximization property and show that leaky ReLU networks trained on hinge loss
with Gradient Descent (GD) satisfy this property. In contrast to prior work we
do not require near orthogonality conditions on the training data: notably, for
input dimension $d$ and training sample size $n$, while prior work shows
asymptotically optimal error when $d = \Omega(n^2 \log n)$, here we require
only $d = \Omega\left(n \log \frac{1}{\epsilon}\right)$ to obtain error within
$\epsilon$ of optimal.

该研究针对二元分类任务，使用带有折线损失的两层泄漏整流线性单元网络，研究了良性过拟合问题，通过对模型参数的信号噪声比进行特征化，发现高信噪比出现良性过拟合，低信噪比出现有害过拟合，并将良性和非良性过拟合归因于近似边际最大化特性，同时降低了训练数据的正交性要求。