It has been found that Transformer-based language models have the ability to perform basic quantitative reasoning. In this paper, we propose a method for studying how these models internally represent numerical data, and use our proposal to analyze the ALBERT family of language models. Specifically, we extract the learned embeddings these models use to represent tokens that correspond to numbers and ordinals, and subject these embeddings to Principal Component Analysis (PCA). PCA results reveal that ALBERT models of different sizes, trained and initialized separately, consistently learn to use the axes of greatest variation to represent the approximate ordering of various numerical concepts. Numerals and their textual counterparts are represented in separate clusters, but increase along the same direction in 2D space. Our findings illustrate that language models, trained purely to model text, can intuit basic mathematical concepts, opening avenues for NLP applications that intersect with quantitative reasoning.

该研究提出了一种方法来研究Transformer语言模型内部如何表示数字数据，并使用该方法分析了ALBERT语言模型系列。通过主成分分析（PCA），我们提取这些模型用于表示数字和序数的令牌的学习嵌入，PCA结果显示不同大小、训练和初始化分开的ALBERT模型一致地学习使用变化最大的轴来表示各种数值概念的近似排序，数字及其文本对应部分分别在不同的簇中表示，但在2D空间中沿着相同的方向增长。我们的发现表明，纯粹用于建模文本的语言模型可以理解基本的数学概念，为与定量推理交叉的NLP应用开辟了新的发展路径。

探索语言模型中的内部数理能力：ALBERT的研究案例