Multimodal information, together with our knowledge, help us to understand
the complex and dynamic world. Large language models (LLM) and large multimodal
models (LMM), however, still struggle to emulate this capability. In this
paper, we present WorldQA, a video understanding dataset designed to push the
boundaries of multimodal world models with three appealing properties: (1)
Multimodal Inputs: The dataset comprises 1007 question-answer pairs and 303
videos, necessitating the analysis of both auditory and visual data for
successful interpretation. (2) World Knowledge: We identify five essential
types of world knowledge for question formulation. This approach challenges
models to extend their capabilities beyond mere perception. (3) Long-Chain
Reasoning: Our dataset introduces an average reasoning step of 4.45, notably
surpassing other videoQA datasets. Furthermore, we introduce WorldRetriever, an
agent designed to synthesize expert knowledge into a coherent reasoning chain,
thereby facilitating accurate responses to WorldQA queries. Extensive
evaluations of 13 prominent LLMs and LMMs reveal that WorldRetriever, although
being the most effective model, achieved only 70% of humanlevel performance in
multiple-choice questions. This finding highlights the necessity for further
advancement in the reasoning and comprehension abilities of models. Our
experiments also yield several key insights. For instance, while humans tend to
perform better with increased frames, current LMMs, including WorldRetriever,
show diminished performance under similar conditions. We hope that WorldQA,our
methodology, and these insights could contribute to the future development of
multimodal world models.

多模态信息和知识对我们理解复杂动态的世界至关重要。本文介绍了一个名为 WorldQA 的视频理解数据集，旨在推动多模态世界模型的发展，并通过引入多种观点挑战模型的能力。通过 WorldRetriever，介绍了一种将专家知识综合整理为连贯推理链条的方法，以便准确回答 WorldQA 的问题。研究还发现了多个关键见解，其中包括模型的推理和理解能力的进一步发展的必要性。希望 WorldQA、方法学以及这些见解可以为多模态世界模型的未来发展做出贡献。