Recently, {\it stochastic momentum} methods have been widely adopted in
training deep neural networks. However, their convergence analysis is still
underexplored at the moment, in particular for non-convex optimization. This
paper fills the gap between practice and theory by developing a basic
convergence analysis of two stochastic momentum methods, namely stochastic
heavy-ball method and the stochastic variant of Nesterov's accelerated gradient
method. We hope that the basic convergence results developed in this paper can
serve the reference to the convergence of stochastic momentum methods and also
serve the baselines for comparison in future development of stochastic momentum
methods. The novelty of convergence analysis presented in this paper is a
unified framework, revealing more insights about the similarities and
differences between different stochastic momentum methods and stochastic
gradient method. The unified framework exhibits a continuous change from the
gradient method to Nesterov's accelerated gradient method and finally the
heavy-ball method incurred by a free parameter, which can help explain a
similar change observed in the testing error convergence behavior for deep
learning. Furthermore, our empirical results for optimizing deep neural
networks demonstrate that the stochastic variant of Nesterov's accelerated
gradient method achieves a good tradeoff (between speed of convergence in
training error and robustness of convergence in testing error) among the three
stochastic methods.

该论文针对随机动量法在非凸优化领域中的收敛性分析不足，通过对两种随机动量法（随机重球法和随机版 Nesterov 加速梯度法）的基本收敛性分析，提出了一种统一框架，展示了它们与随机梯度法之间的相似性和差异性，并在深度学习的测试误差收敛行为中解释了连续变化现象。同时，对深度神经网络的优化实验结果表明，随机版 Nesterov 加速梯度法在训练误差收敛速度和测试误差收敛鲁棒性方面取得了很好的平衡。