Knowledge-based visual question answering (VQA) requires external knowledge
beyond the image to answer the question. Early studies retrieve required
knowledge from explicit knowledge bases (KBs), which often introduces
irrelevant information to the question, hence restricting the performance of
their models. Recent works have sought to use a large language model (i.e.,
GPT-3) as an implicit knowledge engine to acquire the necessary knowledge for
answering. Despite the encouraging results achieved by these methods, we argue
that they have not fully activated the capacity of GPT-3 as the provided input
information is insufficient. In this paper, we present Prophet -- a
conceptually simple framework designed to prompt GPT-3 with answer heuristics
for knowledge-based VQA. Specifically, we first train a vanilla VQA model on a
specific knowledge-based VQA dataset without external knowledge. After that, we
extract two types of complementary answer heuristics from the model: answer
candidates and answer-aware examples. Finally, the two types of answer
heuristics are encoded into the prompts to enable GPT-3 to better comprehend
the task thus enhancing its capacity. Prophet significantly outperforms all
existing state-of-the-art methods on two challenging knowledge-based VQA
datasets, OK-VQA and A-OKVQA, delivering 61.1% and 55.7% accuracies on their
testing sets, respectively.

本论文提出了一种名为 “Prophet” 的框架，采用两个补充答案启发式条目对 GPT-3 进行提示，以提高其理解任务的能力，从而在基于知识的视觉问答方面显著优于所有现有的最先进方法，分别在 OK-VQA 和 A-OKVQA 的测试集上达到了 61.1％和 55.7％的准确率。