Zero-shot Visual Question Answering (VQA) is a prominent vision-language task
that examines both the visual and textual understanding capability of systems
in the absence of training data. Recently, by converting the images into
captions, information across multi-modalities is bridged and Large Language
Models (LLMs) can apply their strong zero-shot generalization capability to
unseen questions. To design ideal prompts for solving VQA via LLMs, several
studies have explored different strategies to select or generate
question-answer pairs as the exemplar prompts, which guide LLMs to answer the
current questions effectively. However, they totally ignore the role of
question prompts. The original questions in VQA tasks usually encounter
ellipses and ambiguity which require intermediate reasoning. To this end, we
present Reasoning Question Prompts for VQA tasks, which can further activate
the potential of LLMs in zero-shot scenarios. Specifically, for each question,
we first generate self-contained questions as reasoning question prompts via an
unsupervised question edition module considering sentence fluency, semantic
integrity and syntactic invariance. Each reasoning question prompt clearly
indicates the intent of the original question. This results in a set of
candidate answers. Then, the candidate answers associated with their confidence
scores acting as answer heuristics are fed into LLMs and produce the final
answer. We evaluate reasoning question prompts on three VQA challenges,
experimental results demonstrate that they can significantly improve the
results of LLMs on zero-shot setting and outperform existing state-of-the-art
zero-shot methods on three out of four data sets. Our source code is publicly
released at https://github.com/ECNU-DASE-NLP/RQP.

通过生成推理问题提示，为零样本情景中的大型语言模型在零样本视觉问答任务中的问题回答问题选择和生成保持完整性、语义合理性和句法不变性，从而显著提高了大型语言模型在零样本情景中的性能。

通过大型语言模型和推理问题提示改进零样本视觉问答

Improving Zero-shot Visual Question Answering via Large Language Models  with Reasoning Question Prompts

This study explores the problem solving capabilities of ChatGPT and its
prospective applications in standardized test preparation, focusing on the GRE
quantitative exam. Prior research has shown great potential for the utilization
of ChatGPT for academic purposes in revolutionizing the approach to studying
across various disciplines. We investigate how ChatGPT performs across various
question types in the GRE quantitative domain, and how modifying question
prompts impacts its accuracy. More specifically this study addressed two
research questions: 1. How does ChatGPT perform in answering GRE-based
quantitative questions across various content areas? 2. How does the accuracy
of ChatGPT vary with modifying the question prompts? The dataset consisting of
100 randomly selected GRE quantitative questions was collected from the ETS
official guide to GRE test preparation. We used quantitative evaluation to
answer our first research question, and t-test to examine the statistical
association between prompt modification and ChatGPT's accuracy. Results show a
statistical improvement in the ChatGPT's accuracy after applying instruction
priming and contextual prompts to the original questions. ChatGPT showed 84%
accuracy with the modified prompts compared to 69% with the original data. The
study discusses the areas where ChatGPT struggled with certain questions and
how modifications can be helpful for preparing for standardized tests like GRE
and provides future directions for prompt modifications.

ChatGPT 在标准化考试准备中的问题解决能力及其潜在应用进行了探索，重点关注 GRE 数学考试。本研究调查了 ChatGPT 在 GRE 数量领域的不同问题类型上的表现，并研究了修改问题提示对其准确性的影响。研究结果表明，在对原始问题使用指令引导和上下文提示后，ChatGPT 的准确性有了统计上的改进。与原始数据 69% 的准确率相比，修改后的提示下 ChatGPT 的准确率达到了 84%。研究讨论了 ChatGPT 在某些问题上的困难之处以及提示修改对于 GRE 等标准化考试准备的帮助，并提供了未来的研究方向。