Humans can readily judge the number of objects in a visual scene, even
without counting, and such a skill has been documented in a variety of animal
species and in babies prior to language development and formal schooling.
Numerical judgments are error-free for small sets, while for larger collections
responses become approximate, with variability increasing proportionally to the
target number. This response pattern is observed for items of all kinds,
despite variation in object features (such as color or shape), suggesting that
our visual number sense relies on abstract representations of numerosity. Here,
we investigated whether generative Artificial Intelligence (AI) models based on
large-scale transformer architectures can reliably name the number of objects
in simple visual stimuli or generate images containing a target number of items
in the 1-10 range. Surprisingly, none of the foundation models considered
performed in a human-like way: They all made striking errors even with small
numbers, the response variability often did not increase in a systematic way,
and the pattern of errors varied with object category. Our findings demonstrate
that advanced AI systems still lack a basic ability that supports an intuitive
understanding of numbers, which in humans is foundational for numeracy and
mathematical development.

通过使用大规模 Transformer 架构的生成式人工智能模型，本研究调查了是否能可靠地命名简单视觉刺激中的物体数量或生成包含 1-10 范围内目标数量物体的图像。令人惊讶的是，所有考虑的基础模型都没有以人类的方式表现出来：即使在小数量下也会出现明显错误，响应的变异性通常没有按系统方式增加，错误的模式也随物体类别而变化。我们的研究结果表明，高级人工智能系统仍然缺乏支持直观理解数字的基本能力，这对人类的数字能力和数学发展至关重要。

大规模生成式人工智能模型缺乏视觉数值感知能力

Large-scale Generative AI Models Lack Visual Number Sense

Numbers are crucial for various real-world domains such as finance,
economics, and science. Thus, understanding and reasoning with numbers are
essential skills for language models to solve different tasks. While different
numerical benchmarks have been introduced in recent years, they are limited to
specific numerical aspects mostly. In this paper, we propose a hierarchical
taxonomy for numerical reasoning skills with more than ten reasoning types
across four levels: representation, number sense, manipulation, and complex
reasoning. We conduct a comprehensive evaluation of state-of-the-art models to
identify reasoning challenges specific to them. Henceforth, we develop a
diverse set of numerical probes employing a semi-automated approach. We focus
on the tabular Natural Language Inference (TNLI) task as a case study and
measure models' performance shifts. Our results show that no model consistently
excels across all numerical reasoning types. Among the probed models, FlanT5
(few-/zero-shot) and GPT-3.5 (few-shot) demonstrate strong overall numerical
reasoning skills compared to other models. Label-flipping probes indicate that
models often exploit dataset artifacts to predict the correct labels.

这篇论文提出了一个层次化的数值推理技能分类法，涵盖四个级别的十多种推理类型。通过对先进模型进行全面评估，并开发了一组多样化的数值探针，论文发现在所有数值推理类型中，FlanT5（无 / 少数据）和 GPT-3.5（少数据）相对其他模型表现出强大的整体数值推理技能。标签颠倒探针表明模型经常利用数据集的特征来预测正确的标签。

探索语言模型的数值推理能力：对表格数据进行综合分析

Exploring the Numerical Reasoning Capabilities of Language Models: A  Comprehensive Analysis on Tabular Data

As a comprehensive indicator of mathematical thinking and intelligence, the
number sense (Dehaene 2011) bridges the induction of symbolic concepts and the
competence of problem-solving. To endow such a crucial cognitive ability to
machine intelligence, we propose a dataset, Machine Number Sense (MNS),
consisting of visual arithmetic problems automatically generated using a
grammar model--And-Or Graph (AOG). These visual arithmetic problems are in the
form of geometric figures: each problem has a set of geometric shapes as its
context and embedded number symbols. Solving such problems is not trivial; the
machine not only has to recognize the number, but also to interpret the number
with its contexts, shapes, and relations (e.g., symmetry) together with proper
operations. We benchmark the MNS dataset using four predominant neural network
models as baselines in this visual reasoning task. Comprehensive experiments
show that current neural-network-based models still struggle to understand
number concepts and relational operations. We show that a simple brute-force
search algorithm could work out some of the problems without context
information. Crucially, taking geometric context into account by an additional
perception module would provide a sharp performance gain with fewer search
steps. Altogether, we call for attention in fusing the classic search-based
algorithms with modern neural networks to discover the essential number
concepts in future research.

为了将关键的认知能力赋予机器智能，本文提出一个新的数据集，即机器数理感知 (MNS)，其由采用 And-Or Graph (AOG) 语法模型自动生成的视觉算术问题组成，提出结合经典搜索算法和现代神经网络的方法，以更好地理解数字概念和关系运算。