Efficient continual learning in humans is enabled by a rich set of
neurophysiological mechanisms and interactions between multiple memory systems.
The brain efficiently encodes information in non-overlapping sparse codes,
which facilitates the learning of new associations faster with controlled
interference with previous associations. To mimic sparse coding in DNNs, we
enforce activation sparsity along with a dropout mechanism which encourages the
model to activate similar units for semantically similar inputs and have less
overlap with activation patterns of semantically dissimilar inputs. This
provides us with an efficient mechanism for balancing the reusability and
interference of features, depending on the similarity of classes across tasks.
Furthermore, we employ sparse coding in a multiple-memory replay mechanism. Our
method maintains an additional long-term semantic memory that aggregates and
consolidates information encoded in the synaptic weights of the working model.
Our extensive evaluation and characteristics analysis show that equipped with
these biologically inspired mechanisms, the model can further mitigate
forgetting.

本文探讨人类的高效持续学习如何通过多个记忆系统的神经生理机制和相互作用来实现，同时介绍了一种基于稀疏编码和多重记忆回放机制的神经网络模型，该模型可以减少遗忘。