The evaluation of Long Video Understanding (LVU) performance poses an
important but challenging research problem. Despite previous efforts, the
existing video understanding benchmarks are severely constrained by several
issues, especially the insufficient lengths of videos, a lack of diversity in
video types and evaluation tasks, and the inappropriateness for evaluating LVU
performances. To address the above problems, we propose a new benchmark, called
MLVU (Multi-task Long Video Understanding Benchmark), for the comprehensive and
in-depth evaluation of LVU. MLVU presents the following critical values: 1) The
substantial and flexible extension of video lengths, which enables the
benchmark to evaluate LVU performance across a wide range of durations. 2) The
inclusion of various video genres, e.g., movies, surveillance footage,
egocentric videos, cartoons, game videos, etc., which reflects the models' LVU
performances in different scenarios. 3) The development of diversified
evaluation tasks, which enables a comprehensive examination of MLLMs' key
abilities in long-video understanding. The empirical study with 20 latest MLLMs
reveals significant room for improvement in today's technique, as all existing
methods struggle with most of the evaluation tasks and exhibit severe
performance degradation when handling longer videos. Additionally, it suggests
that factors such as context length, image-understanding quality, and the
choice of LLM backbone can play critical roles in future advancements. We
anticipate that MLVU will advance the research of long video understanding by
providing a comprehensive and in-depth analysis of MLLMs.

为了解决现有视频理解基准测试中存在的问题，本文提出了一个新的基准测试 MLVU（多任务长视频理解基准测试），包括视频长度的灵活扩展、各种视频类型的包含以及多样化的评估任务，通过对最新 MLLMs 的实证研究，揭示了今天的技术在长视频理解方面仍有改进的空间，提出了未来进展中上下文长度、图像理解质量和 LLM 骨干选择等因素发挥关键作用，预期 MLVU 将通过对 MLLMs 进行全面深入的分析，推动长视频理解的研究。