This paper is on long-term video understanding where the goal is to recognise
human actions over long temporal windows (up to minutes long). In prior work,
long temporal context is captured by constructing a long-term memory bank
consisting of past and future video features which are then integrated into
standard (short-term) video recognition backbones through the use of attention
mechanisms. Two well-known problems related to this approach are the quadratic
complexity of the attention operation and the fact that the whole feature bank
must be stored in memory for inference. To address both issues, we propose an
alternative to attention-based schemes which is based on a low-rank
approximation of the memory obtained using Singular Value Decomposition. Our
scheme has two advantages: (a) it reduces complexity by more than an order of
magnitude, and (b) it is amenable to an efficient implementation for the
calculation of the memory bases in an incremental fashion which does not
require the storage of the whole feature bank in memory. The proposed scheme
matches or surpasses the accuracy achieved by attention-based mechanisms while
being memory-efficient. Through extensive experiments, we demonstrate that our
framework generalises to different architectures and tasks, outperforming the
state-of-the-art in three datasets.

这篇研究论文讨论了长期视频理解的问题，提出了一种基于低秩逼近的方案来解决注意力机制的复杂性和内存存储问题，并通过大量实验验证了该方案在不同架构和任务上的优越性。

MeMSVD: 使用增量 SVD 捕捉长程时域结构

MeMSVD: Long-Range Temporal Structure Capturing Using Incremental SVD

With the success of large language models (LLMs), integrating the vision
model into LLMs to build vision-language foundation models has gained much more
interest recently. However, existing LLM-based large multimodal models (e.g.,
Video-LLaMA, VideoChat) can only take in a limited number of frames for short
video understanding. In this study, we mainly focus on designing an efficient
and effective model for long-term video understanding. Instead of trying to
process more frames simultaneously like most existing work, we propose to
process videos in an online manner and store past video information in a memory
bank. This allows our model to reference historical video content for long-term
analysis without exceeding LLMs' context length constraints or GPU memory
limits. Our memory bank can be seamlessly integrated into current multimodal
LLMs in an off-the-shelf manner. We conduct extensive experiments on various
video understanding tasks, such as long-video understanding, video question
answering, and video captioning, and our model can achieve state-of-the-art
performances across multiple datasets. Code available at
this https URL

通过在在线方式处理视频并将过去的视频信息存储在记忆库中，该研究提出了一种用于长期视频理解的高效有效模型，可以超越语言模型的上下文长度限制和 GPU 内存限制，并在多个数据集上实现了最先进的性能。

MA-LMM：用于长期视频理解的增强记忆大型多模态模型

MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video  Understanding

The challenge of long-term video understanding remains constrained by the
efficient extraction of object semantics and the modelling of their
relationships for downstream tasks. Although the CLIP visual features exhibit
discriminative properties for various vision tasks, particularly in object
encoding, they are suboptimal for long-term video understanding. To address
this issue, we present the Attributes-Aware Network (AAN), which consists of
two key components: the Attributes Extractor and a Graph Reasoning block. These
components facilitate the extraction of object-centric attributes and the
modelling of their relationships within the video. By leveraging CLIP features,
AAN outperforms state-of-the-art approaches on two popular action detection
datasets: Charades and Toyota Smarthome Untrimmed datasets.

通过提取物体语义和模拟它们的关系，Attributes-Aware Network（AAN）在长期视频理解方面取得了优越的性能，包括在动作检测数据集 Charades 和 Toyota Smarthome Untrimmed 数据集上的表现优于最先进方法。