This paper proposes a new mean-field framework for over-parameterized deep
neural networks (DNNs), which can be used to analyze neural network training.
In this framework, a DNN is represented by probability measures and functions
over its features (that is, the function values of the hidden units over the
training data) in the continuous limit, instead of the neural network
parameters as most existing studies have done. This new representation
overcomes the degenerate situation where all the hidden units essentially have
only one meaningful hidden unit in each middle layer, and further leads to a
simpler representation of DNNs, for which the training objective can be
reformulated as a convex optimization problem via suitable re-parameterization.
Moreover, we construct a non-linear dynamics called neural feature flow, which
captures the evolution of an over-parameterized DNN trained by Gradient
Descent. We illustrate the framework via the standard DNN and the Residual
Network (Res-Net) architectures. Furthermore, we show, for Res-Net, when the
neural feature flow process converges, it reaches a global minimal solution
under suitable conditions. Our analysis leads to the first global convergence
proof for over-parameterized neural network training with more than $3$ layers
in the mean-field regime.

该研究提出了一种新的均场框架用于超参数化深度神经网络的分析，利用概率分布和在连续极限下的功能来表示 DNN，并通过适当的重新参数化将其训练目标重新表述为凸优化问题，构建了一种称为神经特征流的非线性动力学来捕捉超参数化 DNN 的演化，证明了在均场区域中超参数化神经网络训练具有全局收敛性的首个全局收敛证明。