We propose a novel framework and a solution to tackle the continual learning
(CL) problem with changing network architectures. Most CL methods focus on
adapting a single architecture to a new task/class by modifying its weights.
However, with rapid progress in architecture design, the problem of adapting
existing solutions to novel architectures becomes relevant. To address this
limitation, we propose Heterogeneous Continual Learning (HCL), where a wide
range of evolving network architectures emerge continually together with novel
data/tasks. As a solution, we build on top of the distillation family of
techniques and modify it to a new setting where a weaker model takes the role
of a teacher; meanwhile, a new stronger architecture acts as a student.
Furthermore, we consider a setup of limited access to previous data and propose
Quick Deep Inversion (QDI) to recover prior task visual features to support
knowledge transfer. QDI significantly reduces computational costs compared to
previous solutions and improves overall performance. In summary, we propose a
new setup for CL with a modified knowledge distillation paradigm and design a
quick data inversion method to enhance distillation. Our evaluation of various
benchmarks shows a significant improvement on accuracy in comparison to
state-of-the-art methods over various networks architectures.

本文提出了一个新的解决方案，Heterogeneous Continual Learning (HCL)，它使用知识蒸馏技术和 Quick Deep Inversion (QDI) 方法来实现对不断变化的网络架构的持续学习。在各种基准测试中进行的评估表明，与现有方法相比，在各种网络架构上的准确性都有显着提高。