Recently, integrating video foundation models and large language models to
build a video understanding system can overcome the limitations of specific
pre-defined vision tasks. Yet, existing methods either employ complex
spatial-temporal modules or rely heavily on additional perception models to
extract temporal features for video understanding, and they only perform well
on short videos. For long videos, the computational complexity and memory costs
associated with long-term temporal connections are significantly increased,
posing additional challenges.Taking advantage of the Atkinson-Shiffrin memory
model, with tokens in Transformers being employed as the carriers of memory in
combination with our specially designed memory mechanism, we propose MovieChat
to overcome these challenges. We lift pre-trained multi-modal large language
models for understanding long videos without incorporating additional trainable
temporal modules, employing a zero-shot approach. MovieChat achieves
state-of-the-art performance in long video understanding, along with the
released MovieChat-1K benchmark with 1K long video, 2K temporal grounding
labels, and 14K manual annotations for validation of the effectiveness of our
method. The code along with the dataset can be accessed via the following
this https URL

基于 Atkinson-Shiffrin 记忆模型与 Transformer 中的记忆承载器，通过特殊设计的记忆机制，无需额外的可训练时序模块，使用零 - shot 方法将预训练的多模态大型语言模型应用于理解长视频，提出了 MovieChat，实现了长视频理解的最新性能，并发布了包含 1K 个长视频、2K 个时序对齐标注和 14K 个手动注释验证方法有效性的 MovieChat-1K 基准。

电影聊天 +: 针对长视频问答的问题感知稀疏记忆

MovieChat+: Question-aware Sparse Memory for Long Video Question  Answering

Recently, integrating video foundation models and large language models to
build a video understanding system overcoming the limitations of specific
pre-defined vision tasks. Yet, existing systems can only handle videos with
very few frames. For long videos, the computation complexity, memory cost, and
long-term temporal connection are the remaining challenges. Inspired by
Atkinson-Shiffrin memory model, we develop an memory mechanism including a
rapidly updated short-term memory and a compact thus sustained long-term
memory. We employ tokens in Transformers as the carriers of memory. MovieChat
achieves state-of-the-art performace in long video understanding.

视频理解系统 MovieChat 利用大型语言模型和视频基础模型，通过引入记忆机制解决了处理长视频时的计算复杂性、内存开销和长期时间连接等挑战。