We analyze multi-layer neural networks in the asymptotic regime of
simultaneously (A) large network sizes and (B) large numbers of stochastic
gradient descent training iterations. We rigorously establish the limiting
behavior of the multi-layer neural network output. The limit procedure is valid
for any number of hidden layers and it naturally also describes the limiting
behavior of the training loss. The ideas that we explore are to (a) take the
limits of each hidden layer sequentially and (b) characterize the evolution of
parameters in terms of their initialization. The limit satisfies a system of
deterministic integro-differential equations. The proof uses methods from weak
convergence and stochastic analysis. We show that, under suitable assumptions
on the activation functions and the behavior for large times, the limit neural
network recovers a global minimum (with zero loss for the objective function).

通过确定性的积分微分方程建模，研究网络尺寸与随机梯度下降迭代次数同时较大时多层神经网络的极限行为，证明任何隐藏层数目下的极限行为，并在合适的激活函数和行为的假设下，表明极限神经网络可恢复全局最小值（目标函数无损失）