Knowledge-based visual question answering (KB-VQA) is a challenging task,
which requires the model to leverage external knowledge for comprehending and
answering questions grounded in visual content. Recent studies retrieve the
knowledge passages from external knowledge bases and then use them to answer
questions. However, these retrieved knowledge passages often contain irrelevant
or noisy information, which limits the performance of the model. To address the
challenge, we propose two synergistic models: Knowledge Condensation model and
Knowledge Reasoning model. We condense the retrieved knowledge passages from
two perspectives. First, we leverage the multimodal perception and reasoning
ability of the visual-language models to distill concise knowledge concepts
from retrieved lengthy passages, ensuring relevance to both the visual content
and the question. Second, we leverage the text comprehension ability of the
large language models to summarize and condense the passages into the knowledge
essence which helps answer the question. These two types of condensed knowledge
are then seamlessly integrated into our Knowledge Reasoning model, which
judiciously navigates through the amalgamated information to arrive at the
conclusive answer. Extensive experiments validate the superiority of the
proposed method. Compared to previous methods, our method achieves
state-of-the-art performance on knowledge-based VQA datasets (65.1% on OK-VQA
and 60.1% on A-OKVQA) without resorting to the knowledge produced by GPT-3
(175B).

我们提出两个协同模型：知识浓缩模型和知识推理模型。这两种浓缩的知识被无缝地集成到我们的知识推理模型中，巧妙地导航综合信息以得出最终答案。与以前的方法相比，我们的方法在基于知识的 VQA 数据集上（OK-VQA 的 65.1％，A-OKVQA 的 60.1％）取得了最先进的性能，而不依赖 GPT-3（175B）生成的知识。

基于知识的 VQA 的知识浓缩和推理

Knowledge Condensation and Reasoning for Knowledge-based VQA

This paper is on the problem of Knowledge-Based Visual Question Answering
(KB-VQA). Recent works have emphasized the significance of incorporating both
explicit (through external databases) and implicit (through LLMs) knowledge to
answer questions requiring external knowledge effectively. A common limitation
of such approaches is that they consist of relatively complicated pipelines and
often heavily rely on accessing GPT-3 API. Our main contribution in this paper
is to propose a much simpler and readily reproducible pipeline which, in a
nutshell, is based on efficient in-context learning by prompting LLaMA (1 and
2) using question-informative captions as contextual information. Contrary to
recent approaches, our method is training-free, does not require access to
external databases or APIs, and yet achieves state-of-the-art accuracy on the
OK-VQA and A-OK-VQA datasets. Finally, we perform several ablation studies to
understand important aspects of our method. Our code is publicly available at
this https URL

这篇研究论文介绍了一种基于知识的视觉问答（KB-VQA）问题的方法，通过在上下文中进行高效的学习，使用问题相关的标题作为上下文信息，而无需训练或访问外部数据库或 API，实现了最先进的准确度。

基于知识的视觉问答的简单基准

A Simple Baseline for Knowledge-Based Visual Question Answering

Given an image and an associated textual question, the purpose of
Knowledge-Based Visual Question Answering (KB-VQA) is to provide a correct
answer to the question with the aid of external knowledge bases. Prior KB-VQA
models are usually formulated as a retriever-classifier framework, where a
pre-trained retriever extracts textual or visual information from knowledge
graphs and then makes a prediction among the candidates. Despite promising
progress, there are two drawbacks with existing models. Firstly, modeling
question-answering as multi-class classification limits the answer space to a
preset corpus and lacks the ability of flexible reasoning. Secondly, the
classifier merely consider "what is the answer" without "how to get the
answer", which cannot ground the answer to explicit reasoning paths. In this
paper, we confront the challenge of \emph{explainable open-set} KB-VQA, where
the system is required to answer questions with entities at wild and retain an
explainable reasoning path. To resolve the aforementioned issues, we propose a
new retriever-ranker paradigm of KB-VQA, Graph pATH rankER (GATHER for
brevity). Specifically, it contains graph constructing, pruning, and path-level
ranking, which not only retrieves accurate answers but also provides inference
paths that explain the reasoning process. To comprehensively evaluate our
model, we reformulate the benchmark dataset OK-VQA with manually corrected
entity-level annotations and release it as ConceptVQA. Extensive experiments on
real-world questions demonstrate that our framework is not only able to perform
open-set question answering across the whole knowledge base but provide
explicit reasoning path.

知识引导的视觉问题回答（KB-VQA）通过外部知识库的辅助，为图像和相关的文本问题提供正确答案。本文提出了一种名为 Graph pATH rankER（GATHER）的新型 KB-VQA 框架，通过构建图、修剪和路径级别排序，不仅能够准确检索答案，还提供解释推理过程的路径。通过在真实世界问题上的广泛实验，证明了该框架不仅能够在整个知识库上进行开放式问题回答，还能提供明确的推理路径。