Stochastic gradient descent (SGD) is a popular and efficient method with wide
applications in training deep neural nets and other nonconvex models. While the
behavior of SGD is well understood in the convex learning setting, the existing
theoretical results for SGD applied to nonconvex objective functions are far
from mature. For example, existing results require to impose a nontrivial
assumption on the uniform boundedness of gradients for all iterates encountered
in the learning process, which is hard to verify in practical implementations.
In this paper, we establish a rigorous theoretical foundation for SGD in
nonconvex learning by showing that this boundedness assumption can be removed
without affecting convergence rates. In particular, we establish sufficient
conditions for almost sure convergence as well as optimal convergence rates for
SGD applied to both general nonconvex objective functions and
gradient-dominated objective functions. A linear convergence is further derived
in the case with zero variances.

本文研究证明了随机梯度下降在非凸学习中，无需统一梯度有界性假设也能达到最优收敛率的情况，并在一定程度上对于一般非凸目标函数和梯度主导的目标函数实现了几乎必然收敛。特别地，在方差为零的情况下可以得到线性收敛。