As deep learning models exponentially increase in size, optimizers such as
Adam encounter significant memory consumption challenges due to the storage of
first and second moment data. Current memory-efficient methods like Adafactor
and CAME often compromise accuracy with their matrix factorization techniques.
Addressing this, we introduce Adapprox, a novel approach that employs
randomized low-rank matrix approximation for a more effective and accurate
approximation of Adam's second moment. Adapprox features an adaptive rank
selection mechanism, finely balancing accuracy and memory efficiency, and
includes an optional cosine similarity guidance strategy to enhance stability
and expedite convergence. In GPT-2 training and downstream tasks, Adapprox
surpasses AdamW by achieving 34.5% to 49.9% and 33.8% to 49.9% memory savings
for the 117M and 345M models, respectively, with the first moment enabled, and
further increases these savings without the first moment. Besides, it enhances
convergence speed and improves downstream task performance relative to its
counterparts.

Adapprox 是一种使用随机低秩矩阵近似来更有效准确地近似 Adam 的二次矩的新方法，在 GPT-2 训练和下游任务中，Adapprox 相比 AdamW 能够实现 34.5％到 49.9％和 33.8％到 49.9％的内存节省，并且它通过提高收敛速度和改进下游任务性能相对于其他方法。