We present a theory for the previously unexplained divergent behavior noticed
in the training of large language models. We argue that the phenomenon is an
artifact of the dominant optimization algorithm used for training, called Adam.
We observe that Adam can enter a state in which the parameter update vector has
a relatively large norm and is essentially uncorrelated with the direction of
descent on the training loss landscape, leading to divergence. This artifact is
more likely to be observed in the training of a deep model with a large batch
size, which is the typical setting of large-scale language model training. To
argue the theory, we present observations from the training runs of the
language models of different scales: 7 billion, 30 billion, 65 billion, and 546
billion parameters.

本文提出了一个关于大型语言模型训练中先前无法解释的发散行为的理论。该理论说明这一现象是优化算法 Adam 的副作用。通过观察该算法的参数更新向量和训练损失方向的相关性，我们得出这一结论，并给出了不同规模语言模型的训练观察结果。