Visual Grounding (VG) in VQA refers to a model's proclivity to infer answers
based on question-relevant image regions. Conceptually, VG identifies as an
axiomatic requirement of the VQA task. In practice, however, DNN-based VQA
models are notorious for bypassing VG by way of shortcut (SC) learning without
suffering obvious performance losses in standard benchmarks. To uncover the
impact of SC learning, Out-of-Distribution (OOD) tests have been proposed that
expose a lack of VG with low accuracy. These tests have since been at the
center of VG research and served as basis for various investigations into VG's
impact on accuracy. However, the role of VG in VQA still remains not fully
understood and has not yet been properly formalized.
In this work, we seek to clarify VG's role in VQA by formalizing it on a
conceptual level. We propose a novel theoretical framework called "Visually
Grounded Reasoning" (VGR) that uses the concepts of VG and Reasoning to
describe VQA inference in ideal OOD testing. By consolidating fundamental
insights into VG's role in VQA, VGR helps to reveal rampant VG-related SC
exploitation in OOD testing, which explains why the relationship between VG and
OOD accuracy has been difficult to define. Finally, we propose an approach to
create OOD tests that properly emphasize a requirement for VG, and show how to
improve performance on them.

通过形式化的理论框架 “Visually Grounded Reasoning”，本研究揭示了在 VQA 中视觉定位的重要性，指出基于深度神经网络的 VQA 模型通过捷径学习（shortcut learning）绕过视觉定位的情况，并提出了用于测试视觉定位的 Out-of-Distribution 测试方法以及改进性能的方法。