As neural networks are increasingly being applied to real-world applications,
mechanisms to address distributional shift and sequential task learning without
forgetting are critical. Methods incorporating network expansion have shown
promise by naturally adding model capacity for learning new tasks while
simultaneously avoiding catastrophic forgetting. However, the growth in the
number of additional parameters of many of these types of methods can be
computationally expensive at larger scales, at times prohibitively so. Instead,
we propose a simple task-specific feature map transformation strategy for
continual learning, which we call Efficient Feature Transformations (EFTs).
These EFTs provide powerful flexibility for learning new tasks, achieved with
minimal parameters added to the base architecture. We further propose a feature
distance maximization strategy, which significantly improves task prediction in
class incremental settings, without needing expensive generative models. We
demonstrate the efficacy and efficiency of our method with an extensive set of
experiments in discriminative (CIFAR-100 and ImageNet-1K) and generative (LSUN,
CUB-200, Cats) sequences of tasks. Even with low single-digit parameter growth
rates, EFTs can outperform many other continual learning methods in a wide
range of settings.

本文提出了一种称为 EFTs 的简单的任务特定特征映射转换策略，具有强大的灵活性，利用最小的参数增加了基本体系结构，使其在大规模场景下实用；我们还提出了一种特征距离最大化策略来完成类增量设置中的任务预测。通过在较广泛的一系列任务序列中的实验，包括区分性任务和生成任务，证明了该方法的实效性和效率。

用于判别性和生成性连续学习的高效特征变换

Efficient Feature Transformations for Discriminative and Generative  Continual Learning

In this paper we propose an approach to avoiding catastrophic forgetting in
sequential task learning scenarios. Our technique is based on a network
reparameterization that approximately diagonalizes the Fisher Information
Matrix of the network parameters. This reparameterization takes the form of a
factorized rotation of parameter space which, when used in conjunction with
Elastic Weight Consolidation (which assumes a diagonal Fisher Information
Matrix), leads to significantly better performance on lifelong learning of
sequential tasks. Experimental results on the MNIST, CIFAR-100, CUB-200 and
Stanford-40 datasets demonstrate that we significantly improve the results of
standard elastic weight consolidation, and that we obtain competitive results
when compared to other state-of-the-art in lifelong learning without
forgetting.

通过重新参数化神经网络中的 Fisher 信息矩阵，结合 Elastic Weight Consolidation 技术，实现解决序列任务学习中的灾难性遗忘问题，并在 MNIST、CIFAR-100、CUB-200 和 Stanford-40 数据集上进行实验证明，我们的方法明显改善了标准 Elastic Weight Consolidation 的结果，在不遗忘的连续任务学习中达到了同等竞争水平。

旋转您的网络：更好的权重合并和更少的灾难性遗忘

Rotate your Networks: Better Weight Consolidation and Less Catastrophic  Forgetting

The ability to learn tasks in a sequential fashion is crucial to the
development of artificial intelligence. Neural networks are not, in general,
capable of this and it has been widely thought that catastrophic forgetting is
an inevitable feature of connectionist models. We show that it is possible to
overcome this limitation and train networks that can maintain expertise on
tasks which they have not experienced for a long time. Our approach remembers
old tasks by selectively slowing down learning on the weights important for
those tasks. We demonstrate our approach is scalable and effective by solving a
set of classification tasks based on the MNIST hand written digit dataset and
by learning several Atari 2600 games sequentially.

研究了神经网络模型中遗忘问题的解决办法并提出了一种基于学习速率调整方法的连续学习模型，实现了在多个任务上获取并保留专业知识的能力。