Pre-trained large language models (LLMs) exhibit impressive mathematical
reasoning capabilities, yet how they compute basic arithmetic, such as
addition, remains unclear. This paper shows that pre-trained LLMs add numbers
using Fourier features -- dimensions in the hidden state that represent numbers
via a set of features sparse in the frequency domain. Within the model, MLP and
attention layers use Fourier features in complementary ways: MLP layers
primarily approximate the magnitude of the answer using low-frequency features,
while attention layers primarily perform modular addition (e.g., computing
whether the answer is even or odd) using high-frequency features. Pre-training
is crucial for this mechanism: models trained from scratch to add numbers only
exploit low-frequency features, leading to lower accuracy. Introducing
pre-trained token embeddings to a randomly initialized model rescues its
performance. Overall, our analysis demonstrates that appropriate pre-trained
representations (e.g., Fourier features) can unlock the ability of Transformers
to learn precise mechanisms for algorithmic tasks.

该研究表明，预训练的大型语言模型使用傅里叶特征进行数字加法，其中 MLP 层主要利用低频特征近似答案的幅度，而注意力层主要利用高频特征进行模块化加法（例如计算答案是奇数还是偶数）。预训练对此机制至关重要，从头开始训练的模型只利用低频特征，导致准确性较低。引入预训练的标记嵌入到随机初始化的模型中可以提高其性能。总的来说，我们的分析表明，适当的预训练表示（例如傅里叶特征）可以为 Transformer 学习算法任务的精确机制。