Text-to-image generative models are capable of producing high-quality images
that often faithfully depict concepts described using natural language. In this
work, we comprehensively evaluate a range of text-to-image models on numerical
reasoning tasks of varying difficulty, and show that even the most advanced
models have only rudimentary numerical skills. Specifically, their ability to
correctly generate an exact number of objects in an image is limited to small
numbers, it is highly dependent on the context the number term appears in, and
it deteriorates quickly with each successive number. We also demonstrate that
models have poor understanding of linguistic quantifiers (such as "a few" or
"as many as"), the concept of zero, and struggle with more advanced concepts
such as partial quantities and fractional representations. We bundle prompts,
generated images and human annotations into GeckoNum, a novel benchmark for
evaluation of numerical reasoning.

我们综合评估了一系列文本到图像生成模型在不同难度的数值推理任务上的性能，并展示即使是最先进的模型在数学技能方面也较为基础，特别是它们在生成图像中正确表达准确数量对象的能力仅限于较小的数字，并高度依赖于数字术语所出现的上下文，并且随着每个连续的数字，能力迅速恶化。我们还展示了模型对语言量化词（如 “几个” 或 “尽可能多”）的理解能力较差，对零的概念有困难，并且在部分数量和分数表示等更高级的概念方面存在困难。我们将提示、生成的图像和人工注释捆绑在一起，推出了 GeckoNum，一个用于评估数值推理的新基准。

评估文图模型中的数值推理

Evaluating Numerical Reasoning in Text-to-Image Models

Recently, there has been significant progress in teaching language models to
perform step-by-step reasoning to solve complex numerical reasoning tasks.
Chain-of-thoughts prompting (CoT) is by far the state-of-art method for these
tasks. CoT uses language models to perform both reasoning and computation in
the multi-step `thought' process. To disentangle computation from reasoning, we
propose `Program of Thoughts' (PoT), which uses language models (mainly Codex)
to express the reasoning process as a program. The computation is relegated to
an external computer, which executes the generated programs to derive the
answer. We evaluate PoT on five math word problem datasets (GSM, AQuA, SVAMP,
TabMWP, MultiArith) and three financial-QA datasets (FinQA, ConvFinQA, TATQA)
for both few-shot and zero-shot setups. Under both few-shot and zero-shot
settings, PoT can show an average performance gain over CoT by around 12\%
across all the evaluated datasets. By combining PoT with self-consistency
decoding, we can achieve SoTA performance on all math problem datasets and
near-SoTA performance on financial datasets. All of our data and code are
released in
Github\footnote{https://github.com/wenhuchen/Program-of-Thoughts}.

用自然语言生成理解模型将数值推理任务中的计算和推理过程分离成程序，使得计算过程集中于电脑中，从而提高了数学问题和金融问题的解决性能。

思维启发计划：将计算与推理相分离用于数学推理任务

Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks

Existing generative pre-trained language models (e.g., GPT) focus on modeling
the language structure and semantics of general texts. However, those models do
not consider the numerical properties of numbers and cannot perform robustly on
numerical reasoning tasks (e.g., math word problems and measurement
estimation). In this paper, we propose NumGPT, a generative pre-trained model
that explicitly models the numerical properties of numbers in texts.
Specifically, it leverages a prototype-based numeral embedding to encode the
mantissa of the number and an individual embedding to encode the exponent of
the number. A numeral-aware loss function is designed to integrate numerals
into the pre-training objective of NumGPT. We conduct extensive experiments on
four different datasets to evaluate the numeracy ability of NumGPT. The
experiment results show that NumGPT outperforms baseline models (e.g., GPT and
GPT with DICE) on a range of numerical reasoning tasks such as measurement
estimation, number comparison, math word problems, and magnitude
classification. Ablation studies are also conducted to evaluate the impact of
pre-training and model hyperparameters on the performance.

本文提出了 NumGPT，这是一种专门用于数学推理的预训练模型，其采用枚举法嵌入来明确地表示文本中的数字特性，并设计了一种数字感知的损失函数，该模型在评估数字能力时的结果表明，其在测量估计、数字比较、数学问题和数量分类等一系列数学推理任务上优于基线模型（如 GPT 和带 DICE 的 GPT）。