Despite the remarkable performance of video-based large language models
(LLMs), their adversarial threat remains unexplored. To fill this gap, we
propose the first adversarial attack tailored for video-based LLMs by crafting
flow-based multi-modal adversarial perturbations on a small fraction of frames
within a video, dubbed FMM-Attack. Extensive experiments show that our attack
can effectively induce video-based LLMs to generate incorrect answers when
videos are added with imperceptible adversarial perturbations. Intriguingly,
our FMM-Attack can also induce garbling in the model output, prompting
video-based LLMs to hallucinate. Overall, our observations inspire a further
understanding of multi-modal robustness and safety-related feature alignment
across different modalities, which is of great importance for various large
multi-modal models. Our code is available at
this https URL

使用流式多模态对抗性扰动，我们提出了首个专为基于视频的大型语言模型设计的对抗性攻击，有效地诱使此类模型生成错误答案，并引发幻觉。这一研究对多模态稳健性和安全相关特性对齐方面的进一步理解具有重要意义。

FMM-Attack：基于流的多模态视频侵略性攻击对 LLMs

FMM-Attack: A Flow-based Multi-modal Adversarial Attack on Video-based  LLMs

Video Anomaly Detection (VAD) aims to localize abnormal events on the
timeline of long-range surveillance videos. Anomaly-scoring-based methods have
been prevailing for years but suffer from the high complexity of thresholding
and low explanability of detection results. In this paper, we conduct pioneer
research on equipping video-based large language models (VLLMs) in the
framework of VAD, making the VAD model free from thresholds and able to explain
the reasons for the detected anomalies. We introduce a novel network module
Long-Term Context (LTC) to mitigate the incapability of VLLMs in long-range
context modeling. We design a three-phase training method to improve the
efficiency of fine-tuning VLLMs by substantially minimizing the requirements
for VAD data and lowering the costs of annotating instruction-tuning data. Our
trained model achieves the top performance on the anomaly videos of the
UCF-Crime and TAD benchmarks, with the AUC improvements of +3.86\% and +4.96\%,
respectively. More impressively, our approach can provide textual explanations
for detected anomalies.

本文介绍了一种基于视频的大型语言模型在视频异常检测中的应用，该方法不需要阈值，并能解释检测到的异常事件。通过引入长期上下文网络模块和三阶段训练方法，提高了模型的性能并降低了数据需求和标注成本，该方法在 UCF-Crime 和 TAD 基准测试中实现了优异的性能。