Text-to-image generative models are capable of producing high-quality images
that often faithfully depict concepts described using natural language. In this
work, we comprehensively evaluate a range of text-to-image models on numerical
reasoning tasks of varying difficulty, and show that even the most advanced
models have only rudimentary numerical skills. Specifically, their ability to
correctly generate an exact number of objects in an image is limited to small
numbers, it is highly dependent on the context the number term appears in, and
it deteriorates quickly with each successive number. We also demonstrate that
models have poor understanding of linguistic quantifiers (such as "a few" or
"as many as"), the concept of zero, and struggle with more advanced concepts
such as partial quantities and fractional representations. We bundle prompts,
generated images and human annotations into GeckoNum, a novel benchmark for
evaluation of numerical reasoning.

我们综合评估了一系列文本到图像生成模型在不同难度的数值推理任务上的性能，并展示即使是最先进的模型在数学技能方面也较为基础，特别是它们在生成图像中正确表达准确数量对象的能力仅限于较小的数字，并高度依赖于数字术语所出现的上下文，并且随着每个连续的数字，能力迅速恶化。我们还展示了模型对语言量化词（如 “几个” 或 “尽可能多”）的理解能力较差，对零的概念有困难，并且在部分数量和分数表示等更高级的概念方面存在困难。我们将提示、生成的图像和人工注释捆绑在一起，推出了 GeckoNum，一个用于评估数值推理的新基准。