The recent developments in Large Multi-modal Video Models (Video-LMMs) have
significantly enhanced our ability to interpret and analyze video data. Despite
their impressive capabilities, current Video-LMMs have not been evaluated for
anomaly detection tasks, which is critical to their deployment in practical
scenarios e.g., towards identifying deepfakes, manipulated video content,
traffic accidents and crimes. In this paper, we introduce VANE-Bench, a
benchmark designed to assess the proficiency of Video-LMMs in detecting and
localizing anomalies and inconsistencies in videos. Our dataset comprises an
array of videos synthetically generated using existing state-of-the-art
text-to-video generation models, encompassing a variety of subtle anomalies and
inconsistencies grouped into five categories: unnatural transformations,
unnatural appearance, pass-through, disappearance and sudden appearance.
Additionally, our benchmark features real-world samples from existing anomaly
detection datasets, focusing on crime-related irregularities, atypical
pedestrian behavior, and unusual events. The task is structured as a visual
question-answering challenge to gauge the models' ability to accurately detect
and localize the anomalies within the videos. We evaluate nine existing
Video-LMMs, both open and closed sources, on this benchmarking task and find
that most of the models encounter difficulties in effectively identifying the
subtle anomalies. In conclusion, our research offers significant insights into
the current capabilities of Video-LMMs in the realm of anomaly detection,
highlighting the importance of our work in evaluating and improving these
models for real-world applications. Our code and data is available at
this https URL

这篇论文介绍了一种用于评估多模态视频模型在异常检测任务中的能力的基准测试方法 VANE-Bench。我们使用合成数据和真实世界样本，评估了九种不同的视频模型在检测和定位视频中的异常情况方面的表现，并发现大部分模型在准确识别微妙异常方面存在困难。此研究为多模态视频模型在异常检测领域的当前能力提供了重要见解，并强调了我们的工作在评估和改进这些模型以用于实际应用中的重要性。

VANE-Bench: 用于会话式 LMM 的视频异常评估基准

VANE-Bench: Video Anomaly Evaluation Benchmark for Conversational LMMs

Recent advancements in Large Language Models (LLMs) have led to the
development of Video Large Multi-modal Models (Video-LMMs) that can handle a
wide range of video understanding tasks. These models have the potential to be
deployed in real-world applications such as robotics, AI assistants, medical
imaging, and autonomous vehicles. The widespread adoption of Video-LMMs in our
daily lives underscores the importance of ensuring and evaluating their robust
performance in mirroring human-like reasoning and interaction capabilities in
complex, real-world contexts. However, existing benchmarks for Video-LMMs
primarily focus on general video comprehension abilities and neglect assessing
their reasoning capabilities over complex videos in the real-world context, and
robustness of these models through the lens of user prompts as text queries. In
this paper, we present the Complex Video Reasoning and Robustness Evaluation
Suite (CVRR-ES), a novel benchmark that comprehensively assesses the
performance of Video-LMMs across 11 diverse real-world video dimensions. We
evaluate 9 recent models, including both open-source and closed-source
variants, and find that most of the Video-LMMs, {especially open-source ones,}
struggle with robustness and reasoning when dealing with complex videos. Based
on our analysis, we develop a training-free Dual-Step Contextual Prompting
(DSCP) technique to enhance the performance of existing Video-LMMs. Our
findings provide valuable insights for building the next generation of
human-centric AI systems with advanced robustness and reasoning capabilities.
Our dataset and code are publicly available at:
this https URL

通过开发一种复杂视频推理和鲁棒性评估套件（CVRR-ES），该研究综合评估了多种最新 Video-LMMs 模型在 11 个不同的现实世界视频维度上的性能，发现大多数 Video-LMMs 模型在处理复杂视频时，特别是开源模型，在鲁棒性和推理能力方面存在困难。此外，作者还提出了一个训练免费的双步骤上下文提示（DSCP）技术以提升现有 Video-LMMs 模型的性能，并为构建具有先进鲁棒性和推理能力的下一代以人为中心的人工智能系统提供了有价值的见解。