Knowledge-Intensive Visual Question Answering (KI-VQA) refers to answering a
question about an image whose answer does not lie in the image. This paper
presents a new pipeline for KI-VQA tasks, consisting of a retriever and a
reader. First, we introduce DEDR, a symmetric dual encoding dense retrieval
framework in which documents and queries are encoded into a shared embedding
space using uni-modal (textual) and multi-modal encoders. We introduce an
iterative knowledge distillation approach that bridges the gap between the
representation spaces in these two encoders. Extensive evaluation on two
well-established KI-VQA datasets, i.e., OK-VQA and FVQA, suggests that DEDR
outperforms state-of-the-art baselines by 11.6% and 30.9% on OK-VQA and FVQA,
respectively. Utilizing the passages retrieved by DEDR, we further introduce
MM-FiD, an encoder-decoder multi-modal fusion-in-decoder model, for generating
a textual answer for KI-VQA tasks. MM-FiD encodes the question, the image, and
each retrieved passage separately and uses all passages jointly in its decoder.
Compared to competitive baselines in the literature, this approach leads to
5.5% and 8.5% improvements in terms of question answering accuracy on OK-VQA
and FVQA, respectively.

本研究提出一种将检索器和阅读器相结合的新 KI-VQA 流程，并通过 DEDR 和 MM-FiD 两种方法进行知识密集型视觉问答 (KI-VQA) 任务，在两个著名的 KI-VQA 数据集 OK-VQA 和 FVQA 上进行了广泛评估，证明了该方法在准确率上优于现有技术。