Adaptive methods such as Adam and RMSProp are widely used in deep learning but are not well understood. In this paper, we seek a crisp, clean and precise characterization of their behavior in nonconvex settings. To this end, we first provide a novel view of adaptive methods as preconditioned SGD, where the preconditioner is estimated in an online manner. By studying the preconditioner on its own, we elucidate its purpose: it rescales the stochastic gradient noise to be isotropic near stationary points, which helps escape saddle points. Furthermore, we show that adaptive methods can efficiently estimate the aforementioned preconditioner. By gluing together these two components, we provide the first (to our knowledge) second-order convergence result for any adaptive method. The key insight from our analysis is that, compared to SGD, adaptive methods escape saddle points faster, and can converge faster overall to second-order stationary points.

该论文研究了深度学习中广泛使用的自适应方法，如Adam和RMSProp，将它们视为预处理的随机梯度下降算法，并提出了新的观点，旨在精确地描述它们在非凸情况下的行为和性能，并证明了它们比传统的SGD算法更快地从鞍点逃脱，并且在总体上更快地收敛到二阶稳定点。

自适应梯度方法逃离鞍点