The Adaptive Momentum Estimation (Adam) algorithm is highly effective in
training various deep learning tasks. Despite this, there's limited theoretical
understanding for Adam, especially when focusing on its vanilla form in
non-convex smooth scenarios with potential unbounded gradients and affine
variance noise. In this paper, we study vanilla Adam under these challenging
conditions. We introduce a comprehensive noise model which governs affine
variance noise, bounded noise and sub-Gaussian noise. We show that Adam can
find a stationary point with a $\mathcal{O}(\text{poly}(\log T)/\sqrt{T})$ rate
in high probability under this general noise model where $T$ denotes total
number iterations, matching the lower rate of stochastic first-order algorithms
up to logarithm factors. More importantly, we reveal that Adam is free of
tuning step-sizes with any problem-parameters, yielding a better adaptation
property than the Stochastic Gradient Descent under the same conditions. We
also provide a probabilistic convergence result for Adam under a generalized
smooth condition which allows unbounded smoothness parameters and has been
illustrated empirically to more accurately capture the smooth property of many
practical objective functions.

在具有潜在无界梯度和仿射方差噪声的非凸光滑场景下，研究了 Adam 算法的理论性质，证明了它能够以高概率在多项式时间复杂度内找到一个稳定点，同时具有较好的自适应性能。