A better understanding of the emergent computation and problem-solving
capabilities of recent large language models is of paramount importance to
further improve them and broaden their applicability. This work investigates
how a language model, trained to predict the next token, can perform arithmetic
computations generalizing beyond training data. Binary addition and
multiplication constitute a good testbed for this purpose, since they require a
very small vocabulary and exhibit relevant input/output discontinuities making
smooth input interpolation ineffective for novel data. We successfully trained
a light language model to learn these tasks and ran a number of experiments to
investigate the extrapolation capabilities and internal information processing.
Our findings support the hypotheses that the language model works as an
Encoding-Regression-Decoding machine where the computation takes place in the
value space once the input token representation is mapped to an appropriate
internal representation.

该研究调查了语言模型在计算预测和问题解决能力方面的应用，以及其在进行算术计算和内部信息处理方面的推广能力。通过使用二进制加法和乘法作为测试样本，该研究成功地训练了一个轻量级语言模型，并进行了一系列实验以探究其外推能力和内部信息处理过程。研究结果支持了该假设，即语言模型在适当的内部表示映射后，计算发生在数值空间中，其工作类似于一个编码 - 回归 - 解码机器。