We consider the momentum stochastic gradient descent scheme (MSGD) and its
continuous-in-time counterpart in the context of non-convex optimization. We
show almost sure exponential convergence of the objective function value for
target functions that are Lipschitz continuous and satisfy the
Polyak-Lojasiewicz inequality on the relevant domain, and under assumptions on
the stochastic noise that are motivated by overparameterized supervised
learning applications. Moreover, we optimize the convergence rate over the set
of friction parameters and show that the MSGD process almost surely converges.

本文研究了非凸优化中动量随机梯度下降 (MSGD) 算法的连续性版本，并证明了在目标函数满足 Lipschitz 连续性和 Polyak-Lojasiewicz 不等式的条件下，MSGD 算法的目标函数极限收敛指数级收敛，同时在给定摩擦参数的情况下，MSGD 过程几乎必定收敛。

关于带有噪声的动量随机梯度下降法在机器学习中的收敛速率

Convergence rates for momentum stochastic gradient descent with noise of  machine learning type

Momentum Stochastic Gradient Descent (MSGD) algorithm has been widely applied
to many nonconvex optimization problems in machine learning, e.g., training
deep neural networks, variational Bayesian inference, and etc. Despite its
empirical success, there is still a lack of theoretical understanding of
convergence properties of MSGD. To fill this gap, we propose to analyze the
algorithmic behavior of MSGD by diffusion approximations for nonconvex
optimization problems with strict saddle points and isolated local optima. Our
study shows that the momentum helps escape from saddle points, but hurts the
convergence within the neighborhood of optima (if without the step size
annealing or momentum annealing). Our theoretical discovery partially
corroborates the empirical success of MSGD in training deep neural networks.

本文通过对非凸优化问题的扩散逼近，分析了 Momentum 随机梯度下降算法的算法行为，发现该算法对于强鞍点的逃逸具有帮助，但在优化器的周围区域内妨碍了收敛（未进行步长退火或动量退火），本文的理论发现部分验证了 MSGD 在训练深度神经网络中的实证成功。