Multimodal Language Language Models (MLLMs) demonstrate the emerging
abilities of "world models" -- interpreting and reasoning about complex
real-world dynamics. To assess these abilities, we posit videos are the ideal
medium, as they encapsulate rich representations of real-world dynamics and
causalities. To this end, we introduce MMWorld, a new benchmark for
multi-discipline, multi-faceted multimodal video understanding. MMWorld
distinguishes itself from previous video understanding benchmarks with two
unique advantages: (1) multi-discipline, covering various disciplines that
often require domain expertise for comprehensive understanding; (2)
multi-faceted reasoning, including explanation, counterfactual thinking, future
prediction, etc. MMWorld consists of a human-annotated dataset to evaluate
MLLMs with questions about the whole videos and a synthetic dataset to analyze
MLLMs within a single modality of perception. Together, MMWorld encompasses
1,910 videos across seven broad disciplines and 69 subdisciplines, complete
with 6,627 question-answer pairs and associated captions. The evaluation
includes 2 proprietary and 10 open-source MLLMs, which struggle on MMWorld
(e.g., GPT-4V performs the best with only 52.3\% accuracy), showing large room
for improvement. Further ablation studies reveal other interesting findings
such as models' different skill sets from humans. We hope MMWorld can serve as
an essential step towards world model evaluation in videos.

多模态语言模型 (MLLMs) 在解释和推理复杂的现实世界动态方面展示出新兴的 “世界模型” 的能力。为了评估这些能力，我们提出视频是理想的媒介，因为它们包含了丰富的现实世界动态和因果关系的表示。为此，我们引入了 MMWorld，一个新的多学科、多方面的多模态视频理解基准。MMWorld 通过两个独特的优势与以前的视频理解基准有所区别：（1）多学科，涵盖各种通常需要领域专业知识才能全面理解的学科；（2）多方面的推理，包括解释、反事实思考、未来预测等。MMWorld 包括一个人工标注的数据集，用于评估 MLLMs 对整个视频的问题，并包括一个用于分析 MLLMs 的单一感知模态的合成数据集。总计 MMWorld 涵盖了 7 个广泛的学科和 69 个子学科的 1,910 个视频，并配有 6,627 个问题 - 答案对和相关字幕。评估包括 2 个专有和 10 个开源的 MLLMs，在 MMWorld 上表现不佳（例如，GPT-4V 的准确率仅为 52.3%），显示了改进的空间。进一步的消融研究揭示了模型与人类不同的技能组合。我们希望 MMWorld 能成为在视频中评估世界模型的关键一步。