Recently, there is a surge in interest surrounding video large language
models (Video LLMs). However, existing benchmarks fail to provide a
comprehensive feedback on the temporal perception ability of Video LLMs. On the
one hand, most of them are unable to distinguish between different temporal
aspects (e.g., speed, direction) and thus cannot reflect the nuanced
performance on these specific aspects. On the other hand, they are limited in
the diversity of task formats (e.g., only multi-choice QA), which hinders the
understanding of how temporal perception performance may vary across different
types of tasks. Motivated by these two problems, we propose the
\textbf{TempCompass} benchmark, which introduces a diversity of temporal
aspects and task formats. To collect high-quality test data, we devise two
novel strategies: (1) In video collection, we construct conflicting videos that
share the same static content but differ in a specific temporal aspect, which
prevents Video LLMs from leveraging single-frame bias or language priors. (2)
To collect the task instructions, we propose a paradigm where humans first
annotate meta-information for a video and then an LLM generates the
instruction. We also design an LLM-based approach to automatically and
accurately evaluate the responses from Video LLMs. Based on TempCompass, we
comprehensively evaluate 8 state-of-the-art (SOTA) Video LLMs and 3 Image LLMs,
and reveal the discerning fact that these models exhibit notably poor temporal
perception ability. Our data will be available at
https://github.com/llyx97/TempCompass.

介绍了 TempCompass 基准测试，该基准测试引入了多样化的时间感知和任务格式，全面评估了 8 种最新的视频大语言模型以及 3 种图像大语言模型，并揭示了这些模型在时间感知方面表现出明显的不足。