Outside-Knowledge Visual Question Answering (OK-VQA) is a challenging VQA
task that requires retrieval of external knowledge to answer questions about
images. Recent OK-VQA systems use Dense Passage Retrieval (DPR) to retrieve
documents from external knowledge bases, such as Wikipedia, but with DPR
trained separately from answer generation, introducing a potential limit on the
overall system performance. Instead, we propose a joint training scheme which
includes differentiable DPR integrated with answer generation so that the
system can be trained in an end-to-end fashion. Our experiments show that our
scheme outperforms recent OK-VQA systems with strong DPR for retrieval. We also
introduce new diagnostic metrics to analyze how retrieval and generation
interact. The strong retrieval ability of our model significantly reduces the
number of retrieved documents needed in training, yielding significant benefits
in answer quality and computation required for training.

我们提出了一种联合训练方案，该方案包括可微分的 DPR 集成答案生成，以便系统可以以端到端的方式进行训练。 我们的实验表明，我们的方案优于最近的具有强大 DPR 进行检索的 OK-VQA 系统。 我们还引入了新的诊断指标来分析检索和生成之间的交互作用，我们的模型的强大检索能力显着降低了训练所需的检索文档数量，从而在答案质量和训练所需的计算方面产生了显着的收益。