We investigate the learning dynamics of classifiers in scenarios where classes are separable or classifiers are over-parameterized. In both cases, Empirical Risk Minimization (ERM) results in zero training error. However, there are many global minima with a training error of zero, some of which generalize well and some of which do not. We show that in separable classes scenarios the proportion of "bad" global minima diminishes exponentially with the number of training data n. Our analysis provides bounds and learning curves dependent solely on the density distribution of the true error for the given classifier function set, irrespective of the set's size or complexity (e.g., number of parameters). This observation may shed light on the unexpectedly good generalization of over-parameterized Neural Networks. For the over-parameterized scenario, we propose a model for the density distribution of the true error, yielding learning curves that align with experiments on MNIST and CIFAR-10.

本研究探讨了在可分类场景或分类器过参数化情况下分类器的学习动态，填补了关于全局最优解的泛化能力的研究空白。通过分析，我们证明在可分类场景中，"不良"全局最优解的比例随着训练数据数量的增加呈指数下降，这一发现为过参数化神经网络的良好泛化能力提供了新视角。

重新思考可分类场景和过参数化状态下分类器的泛化能力