Knowledge-based visual question answering (VQA) requires answering questions
with external knowledge in addition to the content of images. One dataset that
is mostly used in evaluating knowledge-based VQA is OK-VQA, but it lacks a gold
standard knowledge corpus for retrieval. Existing work leverage different
knowledge bases (e.g., ConceptNet and Wikipedia) to obtain external knowledge.
Because of varying knowledge bases, it is hard to fairly compare models'
performance. To address this issue, we collect a natural language knowledge
base that can be used for any VQA system. Moreover, we propose a Visual
Retriever-Reader pipeline to approach knowledge-based VQA. The visual retriever
aims to retrieve relevant knowledge, and the visual reader seeks to predict
answers based on given knowledge. We introduce various ways to retrieve
knowledge using text and images and two reader styles: classification and
extraction. Both the retriever and reader are trained with weak supervision.
Our experimental results show that a good retriever can significantly improve
the reader's performance on the OK-VQA challenge. The code and corpus are
provided in this https URL

本篇研究提出了一种基于知识库检索的视觉问答模型，包括视觉知识检索模型和基于分类和提取两种方式的视觉问题回答模型，并探讨了利用文本和图像进行知识检索的多种方法。实验表明，良好的知识检索模型可以显著提高在 OK-VQA 挑战赛上的视觉问题回答模型表现。