Visual question answering (VQA) is known as an AI-complete task as it
requires understanding, reasoning, and inferring about the vision and the
language content. Over the past few years, numerous neural architectures have
been suggested for the VQA problem. However, achieving success in zero-shot VQA
remains a challenge due to its requirement for advanced generalization and
reasoning skills. This study explores the impact of incorporating image
captioning as an intermediary process within the VQA pipeline. Specifically, we
explore the efficacy of utilizing image captions instead of images and
leveraging large language models (LLMs) to establish a zero-shot setting. Since
image captioning is the most crucial step in this process, we compare the
impact of state-of-the-art image captioning models on VQA performance across
various question types in terms of structure and semantics. We propose a
straightforward and efficient question-driven image captioning approach within
this pipeline to transfer contextual information into the question-answering
(QA) model. This method involves extracting keywords from the question,
generating a caption for each image-question pair using the keywords, and
incorporating the question-driven caption into the LLM prompt. We evaluate the
efficacy of using general-purpose and question-driven image captions in the VQA
pipeline. Our study highlights the potential of employing image captions and
harnessing the capabilities of LLMs to achieve competitive performance on GQA
under the zero-shot setting. Our code is available at
https://github.com/ovguyo/captions-in-VQA.

通过在视觉问题回答（VQA）过程中利用图像描述作为中介步骤，建立零样本设置，探索了零样本 VQA 的影响。比较了最先进的图像描述模型对于不同问题类型的 VQA 性能在结构和语义上的影响，并在 VQA 过程中提出了一种简单高效的问题驱动图像描述方法，通过提取问题关键词、为每个图像 - 问题对生成描述，并将问题驱动的描述传输到大型语言模型中。研究发现利用图像描述和大型语言模型的能力，在零样本设置下能够实现竞争性的 GQA 性能。