Training Large Language Models (LLMs) relies heavily on distributed
implementations, employing multiple GPUs to compute stochastic gradients on
model replicas in parallel. However, synchronizing gradients in data parallel
settings induces a communication overhead increasing with the number of
distributed workers, which can impede the efficiency gains of parallelization.
To address this challenge, optimization algorithms reducing inter-worker
communication have emerged, such as local optimization methods used in
Federated Learning. While effective in minimizing communication overhead, these
methods incur significant memory costs, hindering scalability: in addition to
extra momentum variables, if communications are only allowed between multiple
local optimization steps, then the optimizer's states cannot be sharded among
workers. In response, we propose $\textbf{AC}$cumulate while
$\textbf{CO}$mmunicate ($\texttt{ACCO}$), a memory-efficient optimization
algorithm tailored for distributed training of LLMs. $\texttt{ACCO}$ allows to
shard optimizer states across workers, overlaps gradient computations and
communications to conceal communication costs, and accommodates heterogeneous
hardware. Our method relies on a novel technique to mitigate the one-step delay
inherent in parallel execution of gradient computations and communications,
eliminating the need for warmup steps and aligning with the training dynamics
of standard distributed optimization while converging faster in terms of
wall-clock time. We demonstrate the effectiveness of $\texttt{ACCO}$ on several
LLMs training and fine-tuning tasks.

我们提出了一种适用于大规模语言模型分布式训练的高效优化算法，可以减小通信成本，加速收敛速度，并且适应异构硬件。