A main puzzle of deep networks revolves around the absence of overfitting
despite large overparametrization and despite the large capacity demonstrated
by zero training error on randomly labeled data. In this note, we show that the
dynamics associated to gradient descent minimization of nonlinear networks is
topologically equivalent, near the asymptotically stable minima of the
empirical error, to linear gradient system in a quadratic potential with a
degenerate (for square loss) or almost degenerate (for logistic or crossentropy
loss) Hessian. The proposition depends on the qualitative theory of dynamical
systems and is supported by numerical results. Our main propositions extend to
deep nonlinear networks two properties of gradient descent for linear networks,
that have been recently established (1) to be key to their generalization
properties: 1. Gradient descent enforces a form of implicit regularization
controlled by the number of iterations, and asymptotically converges to the
minimum norm solution for appropriate initial conditions of gradient descent.
This implies that there is usually an optimum early stopping that avoids
overfitting of the loss. This property, valid for the square loss and many
other loss functions, is relevant especially for regression. 2. For
classification, the asymptotic convergence to the minimum norm solution implies
convergence to the maximum margin solution which guarantees good classification
error for "low noise" datasets. This property holds for loss functions such as
the logistic and cross-entropy loss independently of the initial conditions.
The robustness to overparametrization has suggestive implications for the
robustness of the architecture of deep convolutional networks with respect to
the curse of dimensionality.

该研究探讨深度网络中的过拟合问题，发现梯度下降在非线性网络中的优化动力学与线性系统是等价的，同时也推广了梯度下降的两个性质到非线性网络中：隐式正则化以及最小范数解的渐近收敛，通过这些性质，可以提高模型的泛化能力，同时在分类任务中也能得到较好的分类误差。