We present a new multimodal question answering challenge, ManyModalQA, in
which an agent must answer a question by considering three distinct modalities:
text, images, and tables. We collect our data by scraping Wikipedia and then
utilize crowdsourcing to collect question-answer pairs. Our questions are
ambiguous, in that the modality that contains the answer is not easily
determined based solely upon the question. To demonstrate this ambiguity, we
construct a modality selector (or disambiguator) network, and this model gets
substantially lower accuracy on our challenge set, compared to existing
datasets, indicating that our questions are more ambiguous. By analyzing this
model, we investigate which words in the question are indicative of the
modality. Next, we construct a simple baseline ManyModalQA model, which, based
on the prediction from the modality selector, fires a corresponding pre-trained
state-of-the-art unimodal QA model. We focus on providing the community with a
new manymodal evaluation set and only provide a fine-tuning set, with the
expectation that existing datasets and approaches will be transferred for most
of the training, to encourage low-resource generalization without large,
monolithic training sets for each new task. There is a significant gap between
our baseline models and human performance; therefore, we hope that this
challenge encourages research in end-to-end modality disambiguation and
multimodal QA models, as well as transfer learning. Code and data available at:
this https URL

本文提出了一个新的多模态问答挑战 ManyModalQA，在这里代理必须考虑三个不同的模态：文本、图像和表格。我们收集数据的方式是爬取维基百科，并利用众包收集问题 - 答案对。我们的问题是有歧义的，因为包含答案的模态不仅仅基于问题就易于确定。我们构建一个模态选择器（或消歧器）网络，通过分析该模型，我们研究了问题中指示模态的哪些单词。接下来，我们构建了一个简单的基线 ManyModalQA 模型，该模型基于模态选择器的预测，触发相应的预训练单模态 QA 模型。我们提供一个新的多模态评估集，只提供一个微调集，在鼓励低资源泛化新任务时，期望现有数据集和方法能被转移。尽管基于基线模型的表现与人类表现之间存在显著差距，但我们希望这一挑战能鼓励研究多模态 QA 模型的端到端消歧和迁移学习。