We propose a new variant of the Adam optimizer [Kingma and Ba, 2014] called
MICROADAM that specifically minimizes memory overheads, while maintaining
theoretical convergence guarantees. We achieve this by compressing the gradient
information before it is fed into the optimizer state, thereby reducing its
memory footprint significantly. We control the resulting compression error via
a novel instance of the classical error feedback mechanism from distributed
optimization [Seide et al., 2014, Alistarh et al., 2018, Karimireddy et al.,
2019] in which the error correction information is itself compressed to allow
for practical memory gains. We prove that the resulting approach maintains
theoretical convergence guarantees competitive to those of AMSGrad, while
providing good practical performance. Specifically, we show that MICROADAM can
be implemented efficiently on GPUs: on both million-scale (BERT) and
billion-scale (LLaMA) models, MicroAdam provides practical convergence
competitive to that of the uncompressed Adam baseline, with lower memory usage
and similar running time. Our code is available at
this https URL

提出了一种名为 MICROADAM 的 Adam 优化器新变种，它专门最小化内存开销，同时保持理论收敛性保证。通过在优化器状态之前压缩梯度信息来显著减少内存占用。使用分布式优化中的经典错误反馈机制来控制压缩误差，并实现实际的内存收益。证明了这种方法具有与 AMSGrad 相媲美的理论收敛性保证，并提供良好的实际性能。在 GPU 上有效实现的 MICROADAM 在百万级（BERT）和十亿级（LLaMA）模型上，与未压缩的 Adam 基准相比，提供了实用的竞争性收敛性，并具有更低的内存使用和类似的运行时间。