We propose a novel deep network architecture for lifelong learning which we
refer to as Dynamically Expandable Network (DEN), that can dynamically decide
its network capacity as it trains on a sequence of tasks, to learn a compact
overlapping knowledge sharing structure among tasks. DEN is efficiently trained
in an online manner by performing selective retraining, dynamically expands
network capacity upon arrival of each task with only the necessary number of
units, and effectively prevents semantic drift by splitting/duplicating units
and timestamping them. We validate DEN on multiple public datasets under
lifelong learning scenarios, on which it not only significantly outperforms
existing lifelong learning methods for deep networks, but also achieves the
same level of performance as the batch counterparts with substantially fewer
number of parameters. Further, the obtained network fine-tuned on all tasks
obtained significantly better performance over the batch models, which shows
that it can be used to estimate the optimal network structure even when all
tasks are available in the first place.

本研究提出了一种名为 DEN (Dynamically Expandable Network) 的深度网络架构，可以在学习一系列任务时动态决定其网络容量，从而学习任务之间的重叠知识共享结构，通过选择性重新训练、动态扩展网络容量和时间戳来有效地防止语义漂移，并在现有公共数据集中验证了 DEN 的有效性。