Off-the-shelf pre-trained language models have become the de facto standard
in NLP pipelines for a multitude of downstream tasks. However, the inability of
these models to properly encode numerals limits their performance on tasks
requiring numeric comprehension. We introduce strategies to semantically prime
numerals in any corpus by generating anchors governed by the distribution of
numerals in said corpus, thereby enabling mathematically grounded
representations of these numeral tokens. We establish the superiority of our
proposed techniques through evaluation on a range of numeracy tasks for both
in-domain (seen) and out-domain (unseen) numerals. Further, we expand our
empirical evaluations to numerals ranging from 1 to 10 billion, a significantly
broader range compared to previous studies of the same nature, and we
demonstrate significant improvements in the mathematical grounding of our
learned embeddings.

通过生成受语料库中数字分布控制的锚点，我们引入了一种在任何语料库中语义化编码数字的策略，从而能够提供这些数字标记的数学基础表示。通过对一系列涉及从 1 到 10 亿的数字的数学任务的评估，我们证明了我们提出的技术的优越性，并展示了我们所学嵌入的数学基础的显著改进。

铺设锚点：语言建模中数字的语义启发

Laying Anchors: Semantically Priming Numerals in Language Modeling

We present a large, multilingual study into how vision constrains linguistic
choice, covering four languages and five linguistic properties, such as verb
transitivity or use of numerals. We propose a novel method that leverages
existing corpora of images with captions written by native speakers, and apply
it to nine corpora, comprising 600k images and 3M captions. We study the
relation between visual input and linguistic choices by training classifiers to
predict the probability of expressing a property from raw images, and find
evidence supporting the claim that linguistic properties are constrained by
visual context across languages. We complement this investigation with a corpus
study, taking the test case of numerals. Specifically, we use existing
annotations (number or type of objects) to investigate the effect of different
visual conditions on the use of numeral expressions in captions, and show that
similar patterns emerge across languages. Our methods and findings both confirm
and extend existing research in the cognitive literature. We additionally
discuss possible applications for language generation.

本研究通过对四种语言中的五种语言特征进行大规模、多语言的调查研究，并给出了一种新颖的方法，通过分析 600k 张图像和 3M 张图像的标题，证明了视觉背景对语言使用的影响，并在数字表达的研究中发现了不同视觉条件下语言表达的相似模式，为语言生成应用提供了参考。

大规模跨语言研究：视觉限制对语言选择描述的影响

A Large-Scale Multilingual Study of Visual Constraints on Linguistic Selection of Descriptions

Existing generative pre-trained language models (e.g., GPT) focus on modeling
the language structure and semantics of general texts. However, those models do
not consider the numerical properties of numbers and cannot perform robustly on
numerical reasoning tasks (e.g., math word problems and measurement
estimation). In this paper, we propose NumGPT, a generative pre-trained model
that explicitly models the numerical properties of numbers in texts.
Specifically, it leverages a prototype-based numeral embedding to encode the
mantissa of the number and an individual embedding to encode the exponent of
the number. A numeral-aware loss function is designed to integrate numerals
into the pre-training objective of NumGPT. We conduct extensive experiments on
four different datasets to evaluate the numeracy ability of NumGPT. The
experiment results show that NumGPT outperforms baseline models (e.g., GPT and
GPT with DICE) on a range of numerical reasoning tasks such as measurement
estimation, number comparison, math word problems, and magnitude
classification. Ablation studies are also conducted to evaluate the impact of
pre-training and model hyperparameters on the performance.

本文提出了 NumGPT，这是一种专门用于数学推理的预训练模型，其采用枚举法嵌入来明确地表示文本中的数字特性，并设计了一种数字感知的损失函数，该模型在评估数字能力时的结果表明，其在测量估计、数字比较、数学问题和数量分类等一系列数学推理任务上优于基线模型（如 GPT 和带 DICE 的 GPT）。