Communication-efficient variants of SGD, specifically local SGD, have
received a great deal of interest in recent years. These approaches compute
multiple gradient steps locally, that is on each worker, before averaging model
parameters, helping relieve the critical communication bottleneck in
distributed deep learning training. Although many variants of these approaches
have been proposed, they can sometimes lag behind state-of-the-art adaptive
optimizers for deep learning. In this work, we investigate if the recent
progress in the emerging area of learned optimizers can potentially close this
gap while remaining communication-efficient. Specifically, we meta-learn how to
perform global updates given an update from local SGD iterations. Our results
demonstrate that learned optimizers can substantially outperform local SGD and
its sophisticated variants while maintaining their communication efficiency.
Learned optimizers can even generalize to unseen and much larger datasets and
architectures, including ImageNet and ViTs, and to unseen modalities such as
language modeling. We therefore demonstrate the potential of learned optimizers
for improving communication-efficient distributed learning.

通过学习优化器的方法，本研究证明了学习优化器在保持通信高效性的同时，能够明显优于本地 SGD 及其衍生变体，甚至在未见过的更大规模数据集、架构（如 ImageNet 和 ViTs）以及语言模型等方面具有推广性，从而展示了学习优化器改进通信高效分布式学习的潜力。