Most transformer-based video encoders are limited to short temporal contexts
due to their quadratic complexity. While various attempts have been made to
extend this context, this has often come at the cost of both conceptual and
computational complexity. We propose to instead re-purpose existing pre-trained
video transformers by simply fine-tuning them to attend to memories derived
non-parametrically from past activations. By leveraging redundancy reduction,
our memory-consolidated vision transformer (MC-ViT) effortlessly extends its
context far into the past and exhibits excellent scaling behavior when learning
from longer videos. In doing so, MC-ViT sets a new state-of-the-art in
long-context video understanding on EgoSchema, Perception Test, and Diving48,
outperforming methods that benefit from orders of magnitude more parameters.

利用现有的预训练视频 Transformer，并通过简单的微调来重复利用过去激活衍生的非参数化记忆，从而扩展其上下文到过去，以实现对长视频的优秀学习表现。

记忆巩固实现长时视频理解

Memory Consolidation Enables Long-Context Video Understanding

During deep sleep and under anaesthesia spontaneous patterns of cortical
activation frequently take the form of slow travelling waves. Slow wave sleep
is an important cognitive state especially because of its relevance for memory
consolidation. However, despite extensive research the exact mechanisms are
still ill-understood. Novel methods such as high speed widefield imaging of
GCamP activity offer new potentials. Here we show how data recorded from
transgenic mice under anesthesia can be processed to analyze sources, sinks and
patterns of flow. To make the best possible use of the data novel means of data
processing are necessary. Therefore, we (1) give a an brief account on
processes that play a role in generating slow waves and demonstrate (2) a novel
approach to characterize its patterns in GCamP recordings. While slow waves are
highly variable, it shows that some are surprisingly similar. To enable
quantitative means of analysis and examine the structure of such prototypical
events we propose a novel approach for the characterization of slow waves: The
Helmholtz-Decomposition of gradient-based Dense Optical Flow of the pixeldense
GCamP contrast (df/f). It allows to detect the sources and sinks of activation
and discern them from global patterns of neural flow. Aggregated features can
be analyzed with variational autoencoders. The results unravel regularities
between slow waves and shows how they relate to the experimental conditions.
The approach reveals a complex topology of different features in latent slow
wave space and identifies prototypical examples for each stage.

用高速广域成像分析 GCamP 活动的数据，从转基因小鼠的麻醉中提取出潜在的慢波源与汇，并通过梯度基础密集光流的赫姆霍兹分解来表征慢波的模式，揭示了慢波与实验条件的关系。

Helmholtz 分解和光流：一种表征 GCamP 记录的新方法

Helmholtz-Decomposition and Optical Flow: A new method to characterize  GCamP recordings

Artificial neural networks face the well-known problem of catastrophic
forgetting. What's worse, the degradation of previously learned skills becomes
more severe as the task sequence increases, known as the long-term catastrophic
forgetting. It is due to two facts: first, as the model learns more tasks, the
intersection of the low-error parameter subspace satisfying for these tasks
becomes smaller or even does not exist; second, when the model learns a new
task, the cumulative error keeps increasing as the model tries to protect the
parameter configuration of previous tasks from interference. Inspired by the
memory consolidation mechanism in mammalian brains with synaptic plasticity, we
propose a confrontation mechanism in which Adversarial Neural Pruning and
synaptic Consolidation (ANPyC) is used to overcome the long-term catastrophic
forgetting issue. The neural pruning acts as long-term depression to prune
task-irrelevant parameters, while the novel synaptic consolidation acts as
long-term potentiation to strengthen task-relevant parameters. During the
training, this confrontation achieves a balance in that only crucial parameters
remain, and non-significant parameters are freed to learn subsequent tasks.
ANPyC avoids forgetting important information and makes the model efficient to
learn a large number of tasks. Specifically, the neural pruning iteratively
relaxes the current task's parameter conditions to expand the common parameter
subspace of the task; the synaptic consolidation strategy, which consists of a
structure-aware parameter-importance measurement and an element-wise parameter
updating strategy, decreases the cumulative error when learning new tasks. The
full source code is available at this https URL

通过在大脑的突触可塑性中受到启发，作者提出了一种对抗神经修剪和突触巩固（ANPyC）机制来处理由于任务序列增加而导致的长期灾难性遗忘问题，其中神经修剪充当长期沮丧来修剪与任务无关的参数，而新颖的突触巩固则充当长期增强来增强与任务相关的参数。

通过对抗性神经修剪和突触巩固克服长期灾难性遗忘

Overcoming Long-term Catastrophic Forgetting through Adversarial Neural  Pruning and Synaptic Consolidation

Despite advances in deep learning, neural networks can only learn multiple
tasks when trained on them jointly. When tasks arrive sequentially, they lose
performance on previously learnt tasks. This phenomenon called catastrophic
forgetting is a fundamental challenge to overcome before neural networks can
learn continually from incoming data. In this work, we derive inspiration from
human memory to develop an architecture capable of learning continuously from
sequentially incoming tasks, while averting catastrophic forgetting.
Specifically, our contributions are: (i) a dual memory architecture emulating
the complementary learning systems (hippocampus and the neocortex) in the human
brain, (ii) memory consolidation via generative replay of past experiences,
(iii) demonstrating advantages of generative replay and dual memories via
experiments, and (iv) improved performance retention on challenging tasks even
for low capacity models. Our architecture displays many characteristics of the
mammalian memory and provides insights on the connection between sleep and
learning.

本研究提出了一种双重记忆架构模拟人类大脑的海马体和新皮层系统，通过经验的生成重放的记忆巩固，避免神经网络在顺序任务中的灾难性遗忘现象，进而实现连续学习并展示出哺乳类记忆的很多特征。