Adaptive Moment Estimation (Adam) is a cornerstone optimization algorithm in deep learning, widely recognized for its flexibility with adaptive learning rates and efficiency in handling large-scale data. However, despite its practical success, the theoretical understanding of Adam's convergence has been constrained by stringent assumptions, such as almost surely bounded stochastic gradients or uniformly bounded gradients, which are more restrictive than those typically required for analyzing stochastic gradient descent (SGD). In this paper, we introduce a novel and comprehensive framework for analyzing the convergence properties of Adam. This framework offers a versatile approach to establishing Adam's convergence. Specifically, we prove that Adam achieves asymptotic (last iterate sense) convergence in both the almost sure sense and the \(L_1\) sense under the relaxed assumptions typically used for SGD, namely \(L\)-smoothness and the ABC inequality. Meanwhile, under the same assumptions, we show that Adam attains non-asymptotic sample complexity bounds similar to those of SGD.

本研究针对现有Adam算法收敛性理论理解的局限性问题，提出了一个新颖的综合框架以分析其收敛特性。研究表明，在放宽的假设下，Adam在几乎确定的意义和\(L_1\)意义上均可实现渐近收敛，同时在样本复杂度上与SGD相似，具有重要的理论与实践意义。

分析Adam算法收敛性的综合框架：弥合与SGD的差距