Large vision-language models (LVLMs) augment language models with visual understanding, enabling multimodal reasoning. However, due to the modality gap between textual and visual data, they often face significant challenges, such as over-reliance on text priors, hallucinations, and limited capacity for complex visual reasoning. Existing benchmarks to evaluate visual reasoning in LVLMs often rely on schematic or synthetic images and on imprecise machine-generated explanations. To bridge the modality gap, we present DrivingVQA, a new benchmark derived from driving theory tests to evaluate visual chain-of-thought reasoning in complex real-world scenarios. It offers 3,931 expert-crafted multiple-choice problems and interleaved explanations grounded with entities relevant to the reasoning process. We leverage this dataset to perform an extensive study of LVLMs' ability to reason about complex visual scenarios. Our experiments reveal that open-source and proprietary LVLMs struggle with visual chain-of-thought reasoning under zero-shot settings. We investigate training strategies that leverage relevant entities to improve visual reasoning. Notably, we observe a performance boost of up to 7\% when reasoning over image tokens of cropped regions tied to these entities.

本研究解决了现有视觉语言模型在复杂视觉推理中面临的挑战，特别是在文本和视觉数据之间的模态差距。提出的新基准DrivingVQA，通过利用驾驶理论测试，评估视觉链思维推理的能力，发现现有模型在零样本设置下的表现不佳，并提出基于相关实体的训练策略以提升推理效果，提升幅度可达7%。

驾驶视觉问答：分析视觉语言模型在现实场景下的视觉链思维推理