Large multimodal models extend the impressive capabilities of large language
models by integrating multimodal understanding abilities. However, it is not
clear how they can emulate the general intelligence and reasoning ability of
humans. As recognizing patterns and abstracting concepts are key to general
intelligence, we introduce PuzzleVQA, a collection of puzzles based on abstract
patterns. With this dataset, we evaluate large multimodal models with abstract
patterns based on fundamental concepts, including colors, numbers, sizes, and
shapes. Through our experiments on state-of-the-art large multimodal models, we
find that they are not able to generalize well to simple abstract patterns.
Notably, even GPT-4V cannot solve more than half of the puzzles. To diagnose
the reasoning challenges in large multimodal models, we progressively guide the
models with our ground truth reasoning explanations for visual perception,
inductive reasoning, and deductive reasoning. Our systematic analysis finds
that the main bottlenecks of GPT-4V are weaker visual perception and inductive
reasoning abilities. Through this work, we hope to shed light on the
limitations of large multimodal models and how they can better emulate human
cognitive processes in the future (Our data and code will be released publicly
at this https URL).

通过对抽象模式的基本概念进行评估，我们发现大型多模态模型在简单抽象模式上无法很好地泛化。通过对大型多模态模型的推理挑战进行系统分析，我们发现主要瓶颈在于较弱的视觉感知和归纳推理能力。

PuzzleVQA: 通过抽象视觉模式诊断语言模型的多模态推理挑战

PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models  with Abstract Visual Patterns

Human visual reasoning is characterized by an ability to identify abstract
patterns from only a small number of examples, and to systematically generalize
those patterns to novel inputs. This capacity depends in large part on our
ability to represent complex visual inputs in terms of both objects and
relations. Recent work in computer vision has introduced models with the
capacity to extract object-centric representations, leading to the ability to
process multi-object visual inputs, but falling short of the systematic
generalization displayed by human reasoning. Other recent models have employed
inductive biases for relational abstraction to achieve systematic
generalization of learned abstract rules, but have generally assumed the
presence of object-focused inputs. Here, we combine these two approaches,
introducing Object-Centric Relational Abstraction (OCRA), a model that extracts
explicit representations of both objects and abstract relations, and achieves
strong systematic generalization in tasks involving complex visual displays.

该研究描述了 Object-Centric Relational Abstraction（OCRA） 模型，其结合了具有抽象表征能力的目标提取和针对关系的归纳偏差，能够从图像输入中提取显式的对象和抽象关系，实现了关于复杂视觉展示的强系统化概括。