Neural networks trained to minimize the logistic (a.k.a. cross-entropy) loss with gradient-based methods are observed to perform well in many supervised classification tasks. Towards understanding this phenomenon, we analyze the training and generalization behavior of infinitely wide two-layer neural networks with homogeneous activations. We show that the limits of the gradient flow on exponentially tailed losses can be fully characterized as a max-margin classifier in a certain non-Hilbertian space of functions. In presence of hidden low-dimensional structures, the resulting margin is independent of the ambiant dimension, which leads to strong generalization bounds. In contrast, training only the output layer implicitly solves a kernel support vector machine, which a priori does not enjoy such an adaptivity. Our analysis of training is non-quantitative in terms of running time but we prove computational guarantees in simplified settings by showing equivalences with online mirror descent. Finally, numerical experiments suggest that our analysis describes well the practical behavior of two-layer neural networks with ReLU activation and confirm the statistical benefits of this implicit bias.

分析了具有同质性激活函数的两层神经网络在无限宽的情况下的训练和泛化行为，并表明在存在低维结构的情况下，梯度流的极限可以完全表征为某些函数空间中的最大间隔分类器，并且具有强的泛化边界，在实践中符合两层神经网络的行为，并证明了其隐式偏差的统计优点。

使用逻辑损失训练的宽两层神经网络的梯度下降的隐含偏见