Artificial neural networks suffer from catastrophic forgetting when they are
sequentially trained on multiple tasks. To overcome this problem, we present a
novel approach based on task-conditioned hypernetworks, i.e., networks that
generate the weights of a target model based on task identity. Continual
learning (CL) is less difficult for this class of models thanks to a simple key
feature: instead of recalling the input-output relations of all previously seen
data, task-conditioned hypernetworks only require rehearsing task-specific
weight realizations, which can be maintained in memory using a simple
regularizer. Besides achieving state-of-the-art performance on standard CL
benchmarks, additional experiments on long task sequences reveal that
task-conditioned hypernetworks display a very large capacity to retain previous
memories. Notably, such long memory lifetimes are achieved in a compressive
regime, when the number of trainable hypernetwork weights is comparable or
smaller than target network size. We provide insight into the structure of
low-dimensional task embedding spaces (the input space of the hypernetwork) and
show that task-conditioned hypernetworks demonstrate transfer learning.
Finally, forward information transfer is further supported by empirical results
on a challenging CL benchmark based on the CIFAR-10/100 image datasets.

本文提出了一种基于任务条件化超网络的新方法，使得连续学习的模型可以通过简单的关键特征记住特定任务的权重实现在记忆中的持久化，并在标准连续学习基准测试上达到了最先进的性能，同时揭示了该方法在迁移学习上的应用前景。