Although adaptive optimization algorithms such as Adam show fast convergence
in many machine learning tasks, this paper identifies a problem of Adam by
analyzing its performance in a simple non-convex synthetic problem, showing
that Adam's fast convergence would possibly lead the algorithm to local
minimums. To address this problem, we improve Adam by proposing a novel
adaptive gradient descent algorithm named AdaX. Unlike Adam that ignores the
past gradients, AdaX exponentially accumulates the long-term gradient
information in the past during training, to adaptively tune the learning rate.
We thoroughly prove the convergence of AdaX in both the convex and non-convex
settings. Extensive experiments show that AdaX outperforms Adam in various
tasks of computer vision and natural language processing and can catch up with
Stochastic Gradient Descent.

本文发现 Adam 算法的快速收敛可能会导致算法陷入局部极小值，为解决这一问题，我们提出了 AdaX 算法，与 Adam 不同之处在于 AdaX 能够在训练过程中积累过去的梯度信息，实现自适应调节学习率，同时本文也证明了 AdaX 算法在凸性和非凸性情景下均能收敛，实验结果表明 AdaX 在计算机视觉和自然语言处理等多个任务上的表现优于 Adam，且能够与随机梯度下降 (SGD) 保持一致。