Large language models (LLM) have become a critical component in many
applications of machine learning. However, standard approaches to training LLM
require a large number of tightly interconnected accelerators, with devices
exchanging gradients and other intermediate states at each optimization step.
While it is difficult to build and maintain a single computing cluster hosting
many accelerators, it might be easier to find several computing clusters each
hosting a smaller number of devices. In this work, we propose a distributed
optimization algorithm, Distributed Low-Communication (DiLoCo), that enables
training of language models on islands of devices that are poorly connected.
The approach is a variant of federated averaging, where the number of inner
steps is large, the inner optimizer is AdamW, and the outer optimizer is
Nesterov momentum. On the widely used C4 dataset, we show that DiLoCo on 8
workers performs as well as fully synchronous optimization while communicating
500 times less. DiLoCo exhibits great robustness to the data distribution of
each worker. It is also robust to resources becoming unavailable over time, and
vice versa, it can seamlessly leverage resources that become available during
training.

利用分布式优化算法 DiLoCo，可以在受连接限制的设备群组中训练语言模型，具有与完全同步优化相媲美的性能，但通信开销降低了 500 倍，并且对于数据分布和资源可用性的变化都具有良好的鲁棒性。