Large language models (LLM) have become a critical component in many
applications of machine learning. However, standard approaches to training LLM
require a large number of tightly interconnected accelerators, with devices
exchanging gradients and other intermediate states at each optimization step.
While it is difficult to build and maintain a single computing cluster hosting
many accelerators, it might be easier to find several computing clusters each
hosting a smaller number of devices. In this work, we propose a distributed
optimization algorithm, Distributed Low-Communication (DiLoCo), that enables
training of language models on islands of devices that are poorly connected.
The approach is a variant of federated averaging, where the number of inner
steps is large, the inner optimizer is AdamW, and the outer optimizer is
Nesterov momentum. On the widely used C4 dataset, we show that DiLoCo on 8
workers performs as well as fully synchronous optimization while communicating
500 times less. DiLoCo exhibits great robustness to the data distribution of
each worker. It is also robust to resources becoming unavailable over time, and
vice versa, it can seamlessly leverage resources that become available during
training.

利用分布式优化算法 DiLoCo，可以在受连接限制的设备群组中训练语言模型，具有与完全同步优化相媲美的性能，但通信开销降低了 500 倍，并且对于数据分布和资源可用性的变化都具有良好的鲁棒性。

DiLoCo: 分布式低通信语言模型训练

DiLoCo: Distributed Low-Communication Training of Language Models

Many resource allocation problems can be formulated as an optimization
problem whose constraints contain sensitive information about participating
users. This paper concerns solving this kind of optimization problem in a
distributed manner while protecting the privacy of user information. Without
privacy considerations, existing distributed algorithms normally consist in a
central entity computing and broadcasting certain public coordination signals
to participating users. However, the coordination signals often depend on user
information, so that an adversary who has access to the coordination signals
can potentially decode information on individual users and put user privacy at
risk. We present a distributed optimization algorithm that preserves
differential privacy, which is a strong notion that guarantees user privacy
regardless of any auxiliary information an adversary may have. The algorithm
achieves privacy by perturbing the public signals with additive noise, whose
magnitude is determined by the sensitivity of the projection operation onto
user-specified constraints. By viewing the differentially private algorithm as
an implementation of stochastic gradient descent, we are able to derive a bound
for the suboptimality of the algorithm. We illustrate the implementation of our
algorithm via a case study of electric vehicle charging. Specifically, we
derive the sensitivity and present numerical simulations for the algorithm.
Through numerical simulations, we are able to investigate various aspects of
the algorithm when being used in practice, including the choice of step size,
number of iterations, and the trade-off between privacy level and
suboptimality.

本文提出了一种分布式优化算法，通过引入差分隐私的方法来保护隐私信息，该算法通过加性噪声扰动公共信号，噪声的大小由投向用户指定约束条件的投影运算的灵敏度决定，将它视为随机梯度下降的一种实现，并通过对其次优性的边界估计来评估算法的性能，同时对电动汽车充电场景进行了数值模拟，评估了算法在实践中的使用效果，包括步长的选择、迭代次数以及隐私级别和子优延迟之间的权衡问题。