Apart from performing arithmetic operations, understanding numbers themselves
is still a challenge for existing language models. Simple generalisations, such
as solving 100+200 instead of 1+2, can substantially affect model performance
(Sivakumar and Moosavi, 2023). Among various techniques, character-level
embeddings of numbers have emerged as a promising approach to improve number
representation. However, this method has limitations as it leaves the task of
aggregating digit representations to the model, which lacks direct supervision
for this process. In this paper, we explore the use of mathematical priors to
compute aggregated digit embeddings and explicitly incorporate these aggregates
into transformer models. This can be achieved either by adding a special token
to the input embeddings or by introducing an additional loss function to
enhance correct predictions. We evaluate the effectiveness of incorporating
this explicit aggregation, analysing its strengths and shortcomings, and
discuss future directions to better benefit from this approach. Our methods,
while simple, are compatible with any pretrained model and require only a few
lines of code, which we have made publicly available.

本研究探索数学先验的使用，通过计算聚合的数字嵌入，将其明确地融入到转换器模型中，并通过添加特殊标记或引入附加损失函数来实现。我们评估了这种明确聚合的有效性，分析了其优势和不足，并讨论了更好地从这种方法中受益的未来方向。我们的方法虽然简单，但与任何预训练模型兼容，只需要几行代码，并已公开可用。

如何利用数字嵌入来表示数字？

How to Leverage Digit Embeddings to Represent Numbers?

Math word problem (MWP) solving faces a dilemma in number representation
learning. In order to avoid the number representation issue and reduce the
search space of feasible solutions, existing works striving for MWP solving
usually replace real numbers with symbolic placeholders to focus on logic
reasoning. However, different from common symbolic reasoning tasks like program
synthesis and knowledge graph reasoning, MWP solving has extra requirements in
numerical reasoning. In other words, instead of the number value itself, it is
the reusable numerical property that matters more in numerical reasoning.
Therefore, we argue that injecting numerical properties into symbolic
placeholders with contextualized representation learning schema can provide a
way out of the dilemma in the number representation issue here. In this work,
we introduce this idea to the popular pre-training language model (PLM)
techniques and build MWP-BERT, an effective contextual number representation
PLM. We demonstrate the effectiveness of our MWP-BERT on MWP solving and
several MWP-specific understanding tasks on both English and Chinese
benchmarks.

本文介绍了一种通过上下文表示学习的方式，将数值属性注入到符号占位符中的方法，以解决数值表示问题。我们将此想法引入到预训练语言模型（PLM）技术中，构建了一种有效的上下文数值表示 PLM（MWP-BERT）。我们在英文和中文基准上证明了 MWP-BERT 在 MWP 求解和几个 MWP 特定理解任务上的有效性。