Knowledge-based visual question answering (KVQA) task aims to answer
questions that require additional external knowledge as well as an
understanding of images and questions. Recent studies on KVQA inject an
external knowledge in a multi-modal form, and as more knowledge is used,
irrelevant information may be added and can confuse the question answering. In
order to properly use the knowledge, this study proposes the following: 1) we
introduce a novel semantic inconsistency measure computed from caption
uncertainty and semantic similarity; 2) we suggest a new external knowledge
assimilation method based on the semantic inconsistency measure and apply it to
integrate explicit knowledge and implicit knowledge for KVQA; 3) the proposed
method is evaluated with the OK-VQA dataset and achieves the state-of-the-art
performance.

本研究提出了一种基于语义不一致度量的新型外部知识同化方法，应用到知识型视觉问答中，旨在减少添加无关信息对问答的干扰并融合含蓄和显式知识，评估结果表明该方法具有最先进性能。

基于不确定性的视觉问答：估计图像与知识库间的语义不一致性

Uncertainty-based Visual Question Answering: Estimating Semantic Inconsistency between Image and Knowledge Base

This paper revisits visual representation in knowledge-based visual question
answering (VQA) and demonstrates that using regional information in a better
way can significantly improve the performance. While visual representation is
extensively studied in traditional VQA, it is under-explored in knowledge-based
VQA even though these two tasks share the common spirit, i.e., rely on visual
input to answer the question. Specifically, we observe that in most
state-of-the-art knowledge-based VQA methods: 1) visual features are extracted
either from the whole image or in a sliding window manner for retrieving
knowledge, and the important relationship within/among object regions is
neglected; 2) visual features are not well utilized in the final answering
model, which is counter-intuitive to some extent. Based on these observations,
we propose a new knowledge-based VQA method REVIVE, which tries to utilize the
explicit information of object regions not only in the knowledge retrieval
stage but also in the answering model. The key motivation is that object
regions and inherent relationship are important for knowledge-based VQA. We
perform extensive experiments on the standard OK-VQA dataset and achieve new
state-of-the-art performance, i.e., 58.0% accuracy, surpassing previous
state-of-the-art method by a large margin (+3.6%). We also conduct detailed
analysis and show the necessity of regional information in different framework
components for knowledge-based VQA. Code is publicly available at
this https URL.

研究表明视觉表征在基于知识的视觉问答（VQA）中具有重要意义，特别是物体区域信息的利用可以显著提高系统性能。作者提出了一种名为 REVIVE 的新型基于知识的 VQA 方法，该方法旨在利用物体区域的显式信息不仅在知识检索阶段，而且在答案模型中提高系统性能，并在标准 OK-VQA 数据集上实现了最新的最高性能。