Despite the non-convex optimization landscape, over-parametrized shallow
networks are able to achieve global convergence under gradient descent. The
picture can be radically different for narrow networks, which tend to get stuck
in badly-generalizing local minima. Here we investigate the cross-over between
these two regimes in the high-dimensional setting, and in particular
investigate the connection between the so-called mean-field/hydrodynamic regime
and the seminal approach of Saad & Solla. Focusing on the case of Gaussian
data, we study the interplay between the learning rate, the time scale, and the
number of hidden units in the high-dimensional dynamics of stochastic gradient
descent (SGD). Our work builds on a deterministic description of SGD in
high-dimensions from statistical physics, which we extend and for which we
provide rigorous convergence rates.

本文探讨了梯度下降在高维中非凸优化领域的应用，通过对浅层网络和窄网络的研究分析了其在全局收敛和局部最小值上的不同表现，研究了随机梯度下降的高维度动态学习中学习率、时间尺度和隐藏单元数量之间的相互作用，并提供了统计物理学中基于确定性描述的 SGD 收敛速率的扩展分析。