Recently, integrating video foundation models and large language models to
build a video understanding system can overcome the limitations of specific
pre-defined vision tasks. Yet, existing methods either employ complex
spatial-temporal modules or rely heavily on additional perception models to
extract temporal features for video understanding, and they only perform well
on short videos. For long videos, the computational complexity and memory costs
associated with long-term temporal connections are significantly increased,
posing additional challenges.Taking advantage of the Atkinson-Shiffrin memory
model, with tokens in Transformers being employed as the carriers of memory in
combination with our specially designed memory mechanism, we propose MovieChat
to overcome these challenges. We lift pre-trained multi-modal large language
models for understanding long videos without incorporating additional trainable
temporal modules, employing a zero-shot approach. MovieChat achieves
state-of-the-art performance in long video understanding, along with the
released MovieChat-1K benchmark with 1K long video, 2K temporal grounding
labels, and 14K manual annotations for validation of the effectiveness of our
method. The code along with the dataset can be accessed via the following
this https URL

基于 Atkinson-Shiffrin 记忆模型与 Transformer 中的记忆承载器，通过特殊设计的记忆机制，无需额外的可训练时序模块，使用零 - shot 方法将预训练的多模态大型语言模型应用于理解长视频，提出了 MovieChat，实现了长视频理解的最新性能，并发布了包含 1K 个长视频、2K 个时序对齐标注和 14K 个手动注释验证方法有效性的 MovieChat-1K 基准。

电影聊天 +: 针对长视频问答的问题感知稀疏记忆

MovieChat+: Question-aware Sparse Memory for Long Video Question  Answering

Image-goal navigation is a challenging task that requires an agent to
navigate to a goal indicated by an image in unfamiliar environments. Existing
methods utilizing diverse scene memories suffer from inefficient exploration
since they use all historical observations for decision-making without
considering the goal-relevant fraction. To address this limitation, we present
MemoNav, a novel memory model for image-goal navigation, which utilizes a
working memory-inspired pipeline to improve navigation performance.
Specifically, we employ three types of navigation memory. The node features on
a map are stored in the short-term memory (STM), as these features are
dynamically updated. A forgetting module then retains the informative STM
fraction to increase efficiency. We also introduce long-term memory (LTM) to
learn global scene representations by progressively aggregating STM features.
Subsequently, a graph attention module encodes the retained STM and the LTM to
generate working memory (WM) which contains the scene features essential for
efficient navigation. The synergy among these three memory types boosts
navigation performance by enabling the agent to learn and leverage
goal-relevant scene features within a topological map. Our evaluation on
multi-goal tasks demonstrates that MemoNav significantly outperforms previous
methods across all difficulty levels in both Gibson and Matterport3D scenes.
Qualitative results further illustrate that MemoNav plans more efficient
routes.

MemoNav 是一种新型的内存模型，利用工作内存类似的流水线来提高图像目标导航的性能，通过三种类型的导航内存以及场景特征的学习和利用，使得代理能够学习并利用与目标相关的场景特征在拓扑图中进行高效导航，实验证明 MemoNav 在吉布森和 Matterport3D 场景的多目标任务中显著优于先前的方法，定性结果进一步说明 MemoNav 规划更高效的路径。

MemoNav: 视觉导航的工作记忆模型

MemoNav: Working Memory Model for Visual Navigation

Real world applications of Reinforcement Learning (RL) are often partially
observable, thus requiring memory. Despite this, partial observability is still
largely ignored by contemporary RL benchmarks and libraries. We introduce
Partially Observable Process Gym (POPGym), a two-part library containing (1) a
diverse collection of 15 partially observable environments, each with multiple
difficulties and (2) implementations of 13 memory model baselines -- the most
in a single RL library. Existing partially observable benchmarks tend to fixate
on 3D visual navigation, which is computationally expensive and only one type
of POMDP. In contrast, POPGym environments are diverse, produce smaller
observations, use less memory, and often converge within two hours of training
on a consumer-grade GPU. We implement our high-level memory API and memory
baselines on top of the popular RLlib framework, providing plug-and-play
compatibility with various training algorithms, exploration strategies, and
distributed training paradigms. Using POPGym, we execute the largest comparison
across RL memory models to date. POPGym is available at
this https URL.

实际应用中的强化学习（RL）通常是部分可观测的，并且需要记忆。本文提出了一个名为 POPGym 的库，包含多个具有多个困难程度的部分可观测环境和 13 种记忆模型基准实现，并在流行的 RLlib 框架上实现了高层内存 API 和记忆基线，具有可插拔性，且执行了迄今为止最大的 RL 内存模型比较。