Video understanding is a crucial next step for multimodal large language
models (MLLMs). To probe specific aspects of video understanding ability,
existing video benchmarks typically require careful video selection based on
the target capability, along with laborious annotation of query-response pairs
to match the specific video content. This process is both challenging and
resource-intensive. In this paper, we propose VideoNIAH (Video Needle In A
Haystack), a benchmark construction framework through synthetic video
generation. VideoNIAH decouples test video content from their query-responses
by inserting unrelated image/text 'needles' into original videos. It generates
annotations solely from these needles, ensuring diversity in video sources and
a variety of query-responses. Additionally, by inserting multiple needles,
VideoNIAH rigorously evaluates the temporal understanding capabilities of
models. We utilized VideoNIAH to compile a video benchmark VNBench, including
tasks such as retrieval, ordering, and counting. VNBench can efficiently
evaluate the fine-grained understanding ability and spatio-temporal modeling
ability of a video model, while also supporting the long-context evaluation.
Additionally, we evaluated recent video-centric multimodal large language
models (MLLMs), both open-source and proprietary, providing a comprehensive
analysis. We found that although proprietary models have significant advantages
over open-source models, all existing video models still perform poorly on
long-distance dependency tasks. VideoNIAH is a simple yet highly scalable
benchmark construction framework, and we believe it will inspire future video
benchmark works. The code and data are available at
this https URL

VideoNIAH 是一个简单但高度可伸缩的基准构建框架，通过合成视频生成，将测试视频内容与查询 - 响应解耦，并通过插入多个不相关的图像 / 文本 ' 针' 来生成注释，从而确保视频来源的多样性和各种查询 - 响应。