Most Outside-Knowledge Visual Question Answering (OK-VQA) systems employ a
two-stage framework that first retrieves external knowledge given the visual
question and then predicts the answer based on the retrieved content. However,
the retrieved knowledge is often inadequate. Retrievals are frequently too
general and fail to cover specific knowledge needed to answer the question.
Also, the naturally available supervision (whether the passage contains the
correct answer) is weak and does not guarantee question relevancy. To address
these issues, we propose an Entity-Focused Retrieval (EnFoRe) model that
provides stronger supervision during training and recognizes question-relevant
entities to help retrieve more specific knowledge. Experiments show that our
EnFoRe model achieves superior retrieval performance on OK-VQA, the currently
largest outside-knowledge VQA dataset. We also combine the retrieved knowledge
with state-of-the-art VQA models, and achieve a new state-of-the-art
performance on OK-VQA.

本论文提出了一种 Entity-Focused Retrieval (EnFoRe) 模型，通过识别问题相关实体来检索更具特定性的知识。实验证明，我们的 EnFoRe 模型可以在当前最大的 OK-VQA 数据集上实现卓越的检索性能，并与最先进的 VQA 模型结合，实现了新的最先进表现。

面向实体的密集语段检索用于外部知识视觉问答

Entity-Focused Dense Passage Retrieval for Outside-Knowledge Visual Question Answering

Outside-Knowledge Visual Question Answering (OK-VQA) is a challenging VQA
task that requires retrieval of external knowledge to answer questions about
images. Recent OK-VQA systems use Dense Passage Retrieval (DPR) to retrieve
documents from external knowledge bases, such as Wikipedia, but with DPR
trained separately from answer generation, introducing a potential limit on the
overall system performance. Instead, we propose a joint training scheme which
includes differentiable DPR integrated with answer generation so that the
system can be trained in an end-to-end fashion. Our experiments show that our
scheme outperforms recent OK-VQA systems with strong DPR for retrieval. We also
introduce new diagnostic metrics to analyze how retrieval and generation
interact. The strong retrieval ability of our model significantly reduces the
number of retrieved documents needed in training, yielding significant benefits
in answer quality and computation required for training.

我们提出了一种联合训练方案，该方案包括可微分的 DPR 集成答案生成，以便系统可以以端到端的方式进行训练。 我们的实验表明，我们的方案优于最近的具有强大 DPR 进行检索的 OK-VQA 系统。 我们还引入了新的诊断指标来分析检索和生成之间的交互作用，我们的模型的强大检索能力显着降低了训练所需的检索文档数量，从而在答案质量和训练所需的计算方面产生了显着的收益。

带有外部知识的检索增强型视觉问答

Retrieval Augmented Visual Question Answering with Outside Knowledge

Outside-knowledge visual question answering (OK-VQA) requires the agent to
comprehend the image, make use of relevant knowledge from the entire web, and
digest all the information to answer the question. Most previous works address
the problem by first fusing the image and question in the multi-modal space,
which is inflexible for further fusion with a vast amount of external
knowledge. In this paper, we call for a paradigm shift for the OK-VQA task,
which transforms the image into plain text, so that we can enable knowledge
passage retrieval, and generative question-answering in the natural language
space. This paradigm takes advantage of the sheer volume of gigantic knowledge
bases and the richness of pre-trained language models. A
Transform-Retrieve-Generate framework (TRiG) framework is proposed, which can
be plug-and-played with alternative image-to-text models and textual knowledge
bases. Experimental results show that our TRiG framework outperforms all
state-of-the-art supervised methods by at least 11.1% absolute margin.

该论文提出了一种利用海量知识库和预训练语言模型的模型，通过将图像转换为纯文本，以进行知识检索和自然语言生成问答，其中使用的框架为 TRiG 框架，并取得了比所有最先进的监督方法多至少 11.1％的绝对优势。