As the capabilities of large multimodal models (LMMs) continue to advance,
evaluating the performance of LMMs emerges as an increasing need. Additionally,
there is an even larger gap in evaluating the advanced knowledge and reasoning
abilities of LMMs in non-English contexts such as Chinese. We introduce CMMMU,
a new Chinese Massive Multi-discipline Multimodal Understanding benchmark
designed to evaluate LMMs on tasks demanding college-level subject knowledge
and deliberate reasoning in a Chinese context. CMMMU is inspired by and
strictly follows the annotation and analysis pattern of MMMU.
CMMMU includes 12k manually collected multimodal questions from college
exams, quizzes, and textbooks, covering six core disciplines: Art & Design,
Business, Science, Health & Medicine, Humanities & Social Science, and Tech &
Engineering, like its companion, MMMU. These questions span 30 subjects and
comprise 39 highly heterogeneous image types, such as charts, diagrams, maps,
tables, music sheets, and chemical structures.
CMMMU focuses on complex perception and reasoning with domain-specific
knowledge in the Chinese context. We evaluate 11 open-source LLMs and one
proprietary GPT-4V(ision). Even GPT-4V only achieves accuracies of 42%,
indicating a large space for improvement. CMMMU will boost the community to
build the next-generation LMMs towards expert artificial intelligence and
promote the democratization of LMMs by providing diverse language contexts.

CMMMU 是一个新的中国大规模多学科多模态理解基准，旨在评估大型多模态模型在中国语境中对要求大学级学科知识和深思熟虑推理的任务上的表现，并推动下一代 LMMs 的发展。

CMMMU: 一个中国的大型多学科多模态理解基准

CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding  Benchmark

Visual language reasoning requires a system to extract text or numbers from
information-dense images like charts or plots and perform logical or arithmetic
reasoning to arrive at an answer. To tackle this task, existing work relies on
either (1) an end-to-end vision-language model trained on a large amount of
data, or (2) a two-stage pipeline where a captioning model converts the image
into text that is further read by another large language model to deduce the
answer. However, the former approach forces the model to answer a complex
question with one single step, and the latter approach is prone to inaccurate
or distracting information in the converted text that can confuse the language
model. In this work, we propose a dual-system for multi-step multimodal
reasoning, which consists of a "System-1" step for visual information
extraction and a "System-2" step for deliberate reasoning. Given an input,
System-2 breaks down the question into atomic sub-steps, each guiding System-1
to extract the information required for reasoning from the image. Experiments
on chart and plot datasets show that our method with a pre-trained System-2
module performs competitively compared to prior work on in- and
out-of-distribution data. By fine-tuning the System-2 module (LLaMA-2 70B) on
only a small amount of data on multi-step reasoning, the accuracy of our method
is further improved and surpasses the best fully-supervised end-to-end approach
by 5.7% and a pipeline approach with FlanPaLM (540B) by 7.5% on a challenging
dataset with human-authored questions.

通过一个包含系统 1 和系统 2 的双系统，本论文提出了一种用于多步骤多模态推理的方法，其中系统 1 用于提取视觉信息，系统 2 用于深入推理。通过实验证明，我们的方法在图表数据集上与之前的工作相比表现出竞争力，在多步骤推理的少量数据上通过微调系统 2 模块（LLaMA-2 70B），我们的方法的准确性得到进一步提升，并在具有人工提出问题的挑战性数据集上超过最佳全监督端到端方法 5.7% 以及具有 FlanPaLM（540B）的流水线方法 7.5%。