Existing benchmarks for visual question answering lack in visual grounding
and complexity, particularly in evaluating spatial reasoning skills. We
introduce FlowVQA, a novel benchmark aimed at assessing the capabilities of
visual question-answering multimodal language models in reasoning with
flowcharts as visual contexts. FlowVQA comprises 2,272 carefully generated and
human-verified flowchart images from three distinct content sources, along with
22,413 diverse question-answer pairs, to test a spectrum of reasoning tasks,
including information localization, decision-making, and logical progression.
We conduct a thorough baseline evaluation on a suite of both open-source and
proprietary multimodal language models using various strategies, followed by an
analysis of directional bias. The results underscore the benchmark's potential
as a vital tool for advancing the field of multimodal modeling, providing a
focused and challenging environment for enhancing model performance in visual
and logical reasoning tasks.

FlowVQA 是一个新颖的基准测试，旨在评估多模态语言模型在使用流程图作为视觉上下文进行推理的能力，其组成包括来自三个不同内容来源的 2,272 张经过精心生成和人工验证的流程图像以及 22,413 个多样化的问答对，以测试一系列推理任务，包括信息定位、决策和逻辑推进。在开源和专有的多模态语言模型上进行了彻底的基准评估，使用了各种策略，并对方向性偏差进行了分析。结果强调了这个基准测试作为推进多模态建模领域的重要工具的潜力，为提高模型在视觉和逻辑推理任务中的性能提供了一个专注和具有挑战性的环境。

FlowVQA：将流程图用于视觉问答中的多模态逻辑映射

FlowVQA: Mapping Multimodal Logic in Visual Question Answering with  Flowcharts

With the development of multimodal large language models (MLLMs) technology,
its general capabilities are increasingly powerful. To evaluate the various
abilities of MLLMs, numerous evaluation systems have emerged. But now there is
still a lack of a comprehensive method to evaluate MLLMs in the tasks related
to flowcharts, which are very important in daily life and work. We propose the
first comprehensive method, FlowCE, to assess MLLMs across various dimensions
for tasks related to flowcharts. It encompasses evaluating MLLMs' abilities in
Reasoning, Localization Recognition, Information Extraction, Logical
Verification, and Summarization on flowcharts. However, we find that even the
GPT4o model achieves only a score of 56.63. Among open-source models,
Phi-3-Vision obtained the highest score of 49.97. We hope that FlowCE can
contribute to future research on multimodal large language models (MLLMs) for
tasks based on flowcharts. We are open-sourcing this project:
https://github.com/360AILAB-NLP/FlowCE

我们提出了综合评估方法 FlowCE，以评估多模态大语言模型（MLLMs）在与流程图相关的任务中的各种能力。