Momentum-based acceleration of stochastic gradient descent (SGD) is widely used in deep learning. We propose the quasi-hyperbolic momentum algorithm (QHM) as an extremely simple alteration of momentum SGD, averaging a plain SGD step with a momentum step. We describe numerous connections to and identities with other algorithms, and we characterize the set of two-state optimization algorithms that QHM can recover. Finally, we propose a QH variant of Adam called QHAdam, and we empirically demonstrate that our algorithms lead to significantly improved training in a variety of settings, including a new state-of-the-art result on WMT16 EN-DE. We hope that these empirical results, combined with the conceptual and practical simplicity of QHM and QHAdam, will spur interest from both practitioners and researchers. PyTorch code is immediately available.

我们提出了一种称为QHM的算法，它是对动量随机梯度下降的一种极其简单的改进，并通过平均简单的SGD步骤和动量步骤来实现。我们考虑了该算法与其他算法之间的联系和等同性，并表征了QHM可以恢复的二状态优化算法集合。最后，我们提出了一种名为QHAdam的QH变体，并在各种情况下进行了实证研究，包括WMT16 EN-DE的新的最先进结果。我们希望这些实证结果，以及QHM和QHAdam的概念和实用的简洁性，将激发从实践者和研究人员的兴趣。

深度学习中的准双曲动量和 Adam