Large Multimodal Models (LMMs) have achieved impressive success in visual
understanding and reasoning, remarkably improving the performance of
mathematical reasoning in a visual context. Yet, a challenging type of visual
math lies in the multimodal graph theory problem, which demands that LMMs
understand the graphical structures accurately and perform multi-step reasoning
on the visual graph. Additionally, exploring multimodal graph theory problems
will lead to more effective strategies in fields like biology, transportation,
and robotics planning. To step forward in this direction, we are the first to
design a benchmark named VisionGraph, used to explore the capabilities of
advanced LMMs in solving multimodal graph theory problems. It encompasses eight
complex graph problem tasks, from connectivity to shortest path problems.
Subsequently, we present a Description-Program-Reasoning (DPR) chain to enhance
the logical accuracy of reasoning processes through graphical structure
description generation and algorithm-aware multi-step reasoning. Our extensive
study shows that 1) GPT-4V outperforms Gemini Pro in multi-step graph
reasoning; 2) All LMMs exhibit inferior perception accuracy for graphical
structures, whether in zero/few-shot settings or with supervised fine-tuning
(SFT), which further affects problem-solving performance; 3) DPR significantly
improves the multi-step graph reasoning capabilities of LMMs and the GPT-4V
(DPR) agent achieves SOTA performance.

大型多模态模型（LMMs）在视觉理解和推理方面取得了令人瞩目的成功，显著提高了数学推理在视觉环境中的性能。然而，多模态图理论问题是一类具有挑战性的视觉数学问题，要求 LMMs 准确理解图形结构并在视觉图上进行多步推理。我们首次设计了一个名为 VisionGraph 的基准，用于探索先进 LMM 在解决多模态图论问题方面的能力。它包括八个复杂的图问题任务，从连通性到最短路径问题。随后，我们提出了一个描述 - 编程 - 推理（DPR）链，在图形结构描述生成和算法感知的多步推理过程中提高了逻辑准确性。我们的广泛研究表明：1）GPT-4V 在多步图推理方面优于 Gemini Pro；2）无论在零 / 少样本设置还是在受监督微调（SFT）中，所有 LMM 在图形结构的感知准确性方面表现较差，这进一步影响了问题解决的性能；3）DPR 显著提高了 LMM 的多步图推理能力，GPT-4V（DPR）代理达到了 SOTA 性能。