Visual Question Answering (VQA) often involves diverse reasoning scenarios
across Vision and Language (V&L). Most prior VQA studies, however, have merely
focused on assessing the model's overall accuracy without evaluating it on
different reasoning cases. Furthermore, some recent works observe that
conventional Chain-of-Thought (CoT) prompting fails to generate effective
reasoning for VQA, especially for complex scenarios requiring multi-hop
reasoning. In this paper, we propose II-MMR, a novel idea to identify and
improve multi-modal multi-hop reasoning in VQA. In specific, II-MMR takes a VQA
question with an image and finds a reasoning path to reach its answer using two
novel language promptings: (i) answer prediction-guided CoT prompt, or (ii)
knowledge triplet-guided prompt. II-MMR then analyzes this path to identify
different reasoning cases in current VQA benchmarks by estimating how many hops
and what types (i.e., visual or beyond-visual) of reasoning are required to
answer the question. On popular benchmarks including GQA and A-OKVQA, II-MMR
observes that most of their VQA questions are easy to answer, simply demanding
"single-hop" reasoning, whereas only a few questions require "multi-hop"
reasoning. Moreover, while the recent V&L model struggles with such complex
multi-hop reasoning questions even using the traditional CoT method, II-MMR
shows its effectiveness across all reasoning cases in both zero-shot and
fine-tuning settings.

本文介绍了一种新的方法 II-MMR，用于在视觉和语言（V&L）中进行多模多跳推理，通过识别和改进 VQA 中的不同推理案例。II-MMR 通过使用两种新的语言提示：（i）答案预测引导的 Chain-of-Thought（CoT）提示，或（ii）知识三元组引导的提示，找到了一个推理路径来回答 VQA 问题。II-MMR 观察到，在包括 GQA 和 A-OKVQA 的流行基准测试中，大多数 VQA 问题容易回答，只需要 “单跳” 推理，而只有很少的问题需要 “多跳” 推理。另外，尽管最近的 V&L 模型在传统的 CoT 方法下难以处理这些复杂的多跳推理问题，但 II-MMR 在零 - shot 和微调设置中展示了其在所有推理案例中的有效性。