Adaptive gradient-based optimizers, particularly Adam, have left their mark
in training large-scale deep learning models. The strength of such optimizers
is that they exhibit fast convergence while being more robust to hyperparameter
choice. However, they often generalize worse than non-adaptive methods. Recent
studies have tied this performance gap to flat minima selection: adaptive
methods tend to find solutions in sharper basins of the loss landscape, which
in turn hurts generalization. To overcome this issue, we propose a new
memory-augmented version of Adam that promotes exploration towards flatter
minima by using a buffer of critical momentum terms during training.
Intuitively, the use of the buffer makes the optimizer overshoot outside the
basin of attraction if it is not wide enough. We empirically show that our
method improves the performance of several variants of Adam on standard
supervised language modelling and image classification tasks.

提出了一种新的 Adam 的记忆增强版本，通过在训练过程中使用关键动量项的缓冲区，推动探索更平坦的最小值，从而提高了标准监督语言建模和图像分类任务的性能。