Multimodal large language models (MLLMs) have shown great potential in
perception and interpretation tasks, but their capabilities in predictive
reasoning remain under-explored. To address this gap, we introduce a novel
benchmark that assesses the predictive reasoning capabilities of MLLMs across
diverse scenarios. Our benchmark targets three important domains: abstract
pattern reasoning, human activity prediction, and physical interaction
prediction. We further develop three evaluation methods powered by large
language model to robustly quantify a model's performance in predicting and
reasoning the future based on multi-visual context. Empirical experiments
confirm the soundness of the proposed benchmark and evaluation methods via
rigorous testing and reveal pros and cons of current popular MLLMs in the task
of predictive reasoning. Lastly, our proposed benchmark provides a standardized
evaluation framework for MLLMs and can facilitate the development of more
advanced models that can reason and predict over complex long sequence of
multimodal input.

多模式大型语言模型在感知和解释任务中显示出巨大潜力，但其在预测推理方面的能力尚未得到充分探索。为填补这一空白，我们引入了一个新颖的基准测试，评估多模式大型语言模型在不同情境下的预测推理能力。我们的基准测试针对三个重要领域：抽象模式推理、人类活动预测和物理交互预测。我们进一步开发了三种由大型语言模型驱动的评估方法，以稳健地量化模型在基于多模态上下文的预测和推理未来方面的性能。经验实验证实了所提出的基准测试和评估方法的合理性，并通过严格测试揭示了当前流行的多模式大型语言模型在预测推理任务中的优缺点。最后，我们提出的基准测试为多模式大型语言模型提供了一个标准化的评估框架，并可以促进开发能够在复杂的长序列多模态输入上进行推理和预测的更先进的模型的发展。