We present a novel task and benchmark for evaluating the ability of
text-to-image(T2I) generation models to produce images that fit commonsense in
real life, which we call Commonsense-T2I. Given two adversarial text prompts
containing an identical set of action words with minor differences, such as "a
lightbulb without electricity" v.s. "a lightbulb with electricity", we evaluate
whether T2I models can conduct visual-commonsense reasoning, e.g. produce
images that fit "the lightbulb is unlit" vs. "the lightbulb is lit"
correspondingly. Commonsense-T2I presents an adversarial challenge, providing
pairwise text prompts along with expected outputs. The dataset is carefully
hand-curated by experts and annotated with fine-grained labels, such as
commonsense type and likelihood of the expected outputs, to assist analyzing
model behavior. We benchmark a variety of state-of-the-art (sota) T2I models
and surprisingly find that, there is still a large gap between image synthesis
and real life photos--even the DALL-E 3 model could only achieve 48.92% on
Commonsense-T2I, and the stable diffusion XL model only achieves 24.92%
accuracy. Our experiments show that GPT-enriched prompts cannot solve this
challenge, and we include a detailed analysis about possible reasons for such
deficiency. We aim for Commonsense-T2I to serve as a high-quality evaluation
benchmark for T2I commonsense checking, fostering advancements in real life
image generation.

我们提出了一个新的任务和基准，用于评估文本到图像生成模型在现实生活中产生符合常识的图像的能力，我们将其称为常识 - T2I。给定两个对抗性的文本提示，其中包含一组相同的动作词但存在细微差异，例如 “没有电的灯泡” 与 “有电的灯泡”，我们评估 T2I 模型是否能进行视觉常识推理，即生成与 “灯泡未点亮” 和 “灯泡点亮” 相符的图像。常识 - T2I 提出了一个对抗性挑战，提供成对的文本提示和预期输出。该数据集由专家精心策划，并用细粒度标签进行注释，例如常识类型和预期输出的可能性，以帮助分析模型行为。我们对各种最先进的 T2I 模型进行了评测，令人惊讶的发现，图像合成与真实生活照片之间仍存在很大差距 —— 即使是 DALL-E 3 模型在常识 - T2I 上的准确率也仅为 48.92％，稳定的扩散 XL 模型仅能达到 24.92％的准确率。我们的实验证明 GPT 强化的提示无法解决这一挑战，并对可能导致此类不足的原因进行了详细分析。我们的目标是将常识 - T2I 作为 T2I 常识检查的高质量评估基准，促进实际生活图像生成的进展。

常识 T2I 挑战：文本到图像生成模型能否理解常识？

Commonsense-T2I Challenge: Can Text-to-Image Generation Models  Understand Commonsense?

Recently, quite a few novel neural architectures were derived to solve math
word problems by predicting expression trees. These architectures varied from
seq2seq models, including encoders leveraging graph relationships combined with
tree decoders. These models achieve good performance on various MWPs datasets
but perform poorly when applied to an adversarial challenge dataset, SVAMP. We
present a novel model MMTM that leverages multi-tasking and multi-decoder
during pre-training. It creates variant tasks by deriving labels using
pre-order, in-order and post-order traversal of expression trees, and uses
task-specific decoders in a multi-tasking framework. We leverage transformer
architectures with lower dimensionality and initialize weights from RoBERTa
model. MMTM model achieves better mathematical reasoning ability and
generalisability, which we demonstrate by outperforming the best state of the
art baseline models from Seq2Seq, GTS, and Graph2Tree with a relative
improvement of 19.4% on an adversarial challenge dataset SVAMP.

提出一种新的神经网络模型 MMTM，利用多任务和多解码器，在预训练期间创建变体任务，使用特定任务的解码器并使用 Transformer 架构。该模型在对抗性挑战数据集 SVAMP 上相对于 Seq2Seq，GTS 和 Graph2Tree 等模型有 19.4％的相对性能提高来取得更好的数学推理能力和普适性。