Knowledge-based visual question answering (KB-VQA) is a challenging task,
which requires the model to leverage external knowledge for comprehending and
answering questions grounded in visual content. Recent studies retrieve the
knowledge passages from external knowledge bases and then use them to answer
questions. However, these retrieved knowledge passages often contain irrelevant
or noisy information, which limits the performance of the model. To address the
challenge, we propose two synergistic models: Knowledge Condensation model and
Knowledge Reasoning model. We condense the retrieved knowledge passages from
two perspectives. First, we leverage the multimodal perception and reasoning
ability of the visual-language models to distill concise knowledge concepts
from retrieved lengthy passages, ensuring relevance to both the visual content
and the question. Second, we leverage the text comprehension ability of the
large language models to summarize and condense the passages into the knowledge
essence which helps answer the question. These two types of condensed knowledge
are then seamlessly integrated into our Knowledge Reasoning model, which
judiciously navigates through the amalgamated information to arrive at the
conclusive answer. Extensive experiments validate the superiority of the
proposed method. Compared to previous methods, our method achieves
state-of-the-art performance on knowledge-based VQA datasets (65.1% on OK-VQA
and 60.1% on A-OKVQA) without resorting to the knowledge produced by GPT-3
(175B).

我们提出两个协同模型：知识浓缩模型和知识推理模型。这两种浓缩的知识被无缝地集成到我们的知识推理模型中，巧妙地导航综合信息以得出最终答案。与以前的方法相比，我们的方法在基于知识的 VQA 数据集上（OK-VQA 的 65.1％，A-OKVQA 的 60.1％）取得了最先进的性能，而不依赖 GPT-3（175B）生成的知识。