Recently, there has been an increasing interest in building question
answering (QA) models that reason across multiple modalities, such as text and
images. However, QA using images is often limited to just picking the answer
from a pre-defined set of options. In addition, images in the real world,
especially in news, have objects that are co-referential to the text, with
complementary information from both modalities. In this paper, we present a new
QA evaluation benchmark with 1,384 questions over news articles that require
cross-media grounding of objects in images onto text. Specifically, the task
involves multi-hop questions that require reasoning over image-caption pairs to
identify the grounded visual object being referred to and then predicting a
span from the news body text to answer the question. In addition, we introduce
a novel multimedia data augmentation framework, based on cross-media knowledge
extraction and synthetic question-answer generation, to automatically augment
data that can provide weak supervision for this task. We evaluate both
pipeline-based and end-to-end pretraining-based multimedia QA models on our
benchmark, and show that they achieve promising performance, while considerably
lagging behind human performance hence leaving large room for future work on
this challenging new task.

本文介绍了一个包含 1,384 个跨媒体对象基于新闻文章的多步骤问题的 QA 评估基准，并提出了一个新颖的多媒体数据增强框架，用于自动增强可以为此任务提供弱监督的数据。在我们的基准上评估了基于流水线和端到端预训练的多媒体 QA 模型，并表明它们实现了有希望的性能，但相当滞后于人类表现，因此为未来此挑战性新任务留下了很大的空间。