Outside-knowledge visual question answering (OK-VQA) requires the agent to
comprehend the image, make use of relevant knowledge from the entire web, and
digest all the information to answer the question. Most previous works address
the problem by first fusing the image and question in the multi-modal space,
which is inflexible for further fusion with a vast amount of external
knowledge. In this paper, we call for a paradigm shift for the OK-VQA task,
which transforms the image into plain text, so that we can enable knowledge
passage retrieval, and generative question-answering in the natural language
space. This paradigm takes advantage of the sheer volume of gigantic knowledge
bases and the richness of pre-trained language models. A
Transform-Retrieve-Generate framework (TRiG) framework is proposed, which can
be plug-and-played with alternative image-to-text models and textual knowledge
bases. Experimental results show that our TRiG framework outperforms all
state-of-the-art supervised methods by at least 11.1% absolute margin.

该论文提出了一种利用海量知识库和预训练语言模型的模型，通过将图像转换为纯文本，以进行知识检索和自然语言生成问答，其中使用的框架为 TRiG 框架，并取得了比所有最先进的监督方法多至少 11.1％的绝对优势。