Continual Learning in artificial neural networks suffers from interference
and forgetting when different tasks are learned sequentially. This paper
introduces the Active Long Term Memory Networks (A-LTM), a model of sequential
multi-task deep learning that is able to maintain previously learned
association between sensory input and behavioral output while acquiring knew
knowledge. A-LTM exploits the non-convex nature of deep neural networks and
actively maintains knowledge of previously learned, inactive tasks using a
distillation loss. Distortions of the learned input-output map are penalized
but hidden layers are free to transverse towards new local optima that are more
favorable for the multi-task objective. We re-frame the McClelland's seminal
Hippocampal theory with respect to Catastrophic Inference (CI) behavior
exhibited by modern deep architectures trained with back-propagation and
inhomogeneous sampling of latent factors across epochs. We present empirical
results of non-trivial CI during continual learning in Deep Linear Networks
trained on the same task, in Convolutional Neural Networks when the task shifts
from predicting semantic to graphical factors and during domain adaptation from
simple to complex environments. We present results of the A-LTM model's ability
to maintain viewpoint recognition learned in the highly controlled iLab-20M
dataset with 10 object categories and 88 camera viewpoints, while adapting to
the unstructured domain of Imagenet with 1,000 object categories.

介绍了 Active Long Term Memory Networks (A-LTM) 模型，该模型能够在序贯多任务深度学习过程中保留以前学习的知识，同时获得新的知识，利用 distillation loss 来主动维护以前学习的信息，并赋予隐藏层向新的多任务目标优化的自由，结果表明 A-LTM 策略可以维持高精度的视角识别，适应复杂的知识领域。