We propose a novel VQA dataset, based on picture stories designed for
educating young children, that aims to facilitate comprehensive evaluation and
characterization of vision-language models on comprehension tasks. Unlike
current VQA datasets that often focus on fact-based memorization and simple
reasoning tasks without principled scientific grounding, we collect data
containing tasks reflecting different levels of comprehension and underlying
cognitive processes, as laid out in Bloom's Taxonomy, a classic framework
widely adopted in education research. The proposed BloomVQA dataset can be
mapped to a hierarchical graph-based representation of visual stories, enabling
automatic data augmentation and novel measures characterizing model consistency
across the underlying taxonomy. We demonstrate graded evaluation and
reliability analysis based on our proposed consistency metrics on
state-of-the-art vision-language models. Our results suggest that, while
current models achieve the most gain on low-level comprehension tasks, they
generally fall short on high-level tasks requiring more advanced comprehension
and cognitive skills, as 38.0% drop in VQA accuracy is observed comparing
lowest and highest level tasks. Furthermore, current models show consistency
patterns misaligned with human comprehension in various scenarios, suggesting
emergent structures of model behaviors.

我们提出了一个新颖的视觉问答（VQA）数据集，基于为教育幼儿而设计的图片故事，旨在通过理解任务对视觉语言模型进行全面评估和特征化。该数据集包含不同层次的理解任务和底层认知过程，可以映射到基于图形的视觉故事表示，实现自动数据增强和模型一致性特征。我们的结果表明，尽管现有模型在低级别理解任务上取得了最大的进步，但在需要更高级理解和认知技能的高级任务上，它们普遍表现不佳，最高级别任务的 VQA 准确度下降了 38.0%。此外，目前的模型在各种情境中与人类理解的一致性模式不匹配，表明模型行为的新出现结构。