While multi-modal large language models (MLLMs) have shown significant
progress on many popular visual reasoning benchmarks, whether they possess
abstract visual reasoning abilities remains an open question. Similar to the
Sudoku puzzles, abstract visual reasoning (AVR) problems require finding
high-level patterns (e.g., repetition constraints) that control the input
shapes (e.g., digits) in a specific task configuration (e.g., matrix). However,
existing AVR benchmarks only considered a limited set of patterns (addition,
conjunction), input shapes (rectangle, square), and task configurations (3 by 3
matrices). To evaluate MLLMs' reasoning abilities comprehensively, we introduce
MARVEL, a multidimensional AVR benchmark with 770 puzzles composed of six core
knowledge patterns, geometric and abstract shapes, and five different task
configurations. To inspect whether the model accuracy is grounded in perception
and reasoning, MARVEL complements the general AVR question with perception
questions in a hierarchical evaluation framework. We conduct comprehensive
experiments on MARVEL with nine representative MLLMs in zero-shot and few-shot
settings. Our experiments reveal that all models show near-random performance
on the AVR question, with significant performance gaps (40%) compared to humans
across all patterns and task configurations. Further analysis of perception
questions reveals that MLLMs struggle to comprehend the visual features
(near-random performance) and even count the panels in the puzzle ( <45%),
hindering their ability for abstract reasoning. We release our entire code and
dataset.

多模态大型语言模型在抽象视觉推理方面呈现出近乎随机的性能，无法理解视觉特征和难以进行抽象推理。

MARVEL: 多维度的可视化评估和学习中的抽象和推理

MARVEL: Multidimensional Abstraction and Reasoning through Visual  Evaluation and Learning

In real-world tasks, there is usually a large amount of unlabeled data and
labeled data. The task of combining the two to learn is known as
semi-supervised learning. Experts can use logical rules to label unlabeled
data, but this operation is costly. The combination of perception and reasoning
has a good effect in processing such semi-supervised tasks with domain
knowledge. However, acquiring domain knowledge and the correction, reduction
and generation of rules remain complex problems to be solved. Rough set theory
is an important method for solving knowledge processing in information systems.
In this paper, we propose a rule general abductive learning by rough set
(RS-ABL). By transforming the target concept and sub-concepts of rules into
information tables, rough set theory is used to solve the acquisition of domain
knowledge and the correction, reduction and generation of rules at a lower
cost. This framework can also generate more extensive negative rules to enhance
the breadth of the knowledge base. Compared with the traditional
semi-supervised learning method, RS-ABL has higher accuracy in dealing with
semi-supervised tasks.

本文提出了一种基于粗集理论的规则一般性诱导方法（Rule General Abductive Learning by Rough Set，RS-ABL），用于解决信息系统中知识处理中的知识获取、纠错、减少、生成和协同处理等问题，并在解决半监督任务时具有更高的准确性。