We present a new pre-training method, Multimodal Inverse Cloze Task, for
Knowledge-based Visual Question Answering about named Entities (KVQAE). KVQAE
is a recently introduced task that consists in answering questions about named
entities grounded in a visual context using a Knowledge Base. Therefore, the
interaction between the modalities is paramount to retrieve information and
must be captured with complex fusion models. As these models require a lot of
training data, we design this pre-training task from existing work in textual
Question Answering. It consists in considering a sentence as a pseudo-question
and its context as a pseudo-relevant passage and is extended by considering
images near texts in multimodal documents. Our method is applicable to
different neural network architectures and leads to a 9% relative-MRR and 15%
relative-F1 gain for retrieval and reading comprehension, respectively, over a
no-pre-training baseline.

本文提出一种基于预训练的多模态逆推理任务方法，用于解决基于知识库的视觉问答，该方法适用于不同的神经网络架构，相比于没有预训练的基准模型，检索和阅读理解的相对 MRR 和相对 F1 分别提高了 9% 和 15%。