Large language models (LLMs) frequently make errors when handling even simple numerical problems, such as comparing two small numbers. A natural hypothesis is that these errors stem from how LLMs represent numbers, and specifically, whether their representations of numbers capture their numeric values. We tackle this question from the observation that LLM errors on numerical tasks are often distributed across \textit{the digits} of the answer rather than normally around \textit{its numeric value}. Through a series of probing experiments and causal interventions, we show that LLMs internally represent numbers with individual circular representations per-digit in base 10. This digit-wise representation, as opposed to a value representation, sheds light on the error patterns of models on tasks involving numerical reasoning and could serve as a basis for future studies on analyzing numerical mechanisms in LLMs.

本研究解决了大型语言模型在处理数字问题时常出现错误的现象。我们通过实验表明，语言模型内部以十进制的每个数字的循环表示法来表示数字，而非简单的数值表示。这一发现有助于理解模型在数字推理任务中出现的错误模式，并为未来研究语言模型中的数字机制提供基础。

语言模型如何以十进制数字表示法编码数字