Stochastic gradient descent is the method of choice for large scale
optimization of machine learning objective functions. Yet, its performance is
greatly variable and heavily depends on the choice of the stepsizes. This has
motivated a large body of research on adaptive stepsizes. However, there is
currently a gap in our theoretical understanding of these methods, especially
in the non-convex setting. In this paper, we start closing this gap: we
theoretically analyze in the convex and non-convex settings a generalized
version of the AdaGrad stepsizes. We show sufficient conditions for these
stepsizes to achieve almost sure asymptotic convergence of the gradients to
zero, proving the first guarantee for generalized AdaGrad stepsizes in the
non-convex setting. Moreover, we show that these stepsizes allow to
automatically adapt to the level of noise of the stochastic gradients in both
the convex and non-convex settings, interpolating between $O(1/T)$ and
$O(1/\sqrt{T})$, up to logarithmic terms.

通过研究广义 AdaGrad 步长在凸和非凸设置中，本文证明了这些步长实现梯度渐近收敛于零的充分条件，从而填补了这些方法理论上的空白。此外，本文表明这些步长允许自动适应随机梯度噪声级别在凸和非凸情况下，实现 O（1/T）到 O（1 / 根号 T）的插值（带有对数项）。