Efforts to overcome catastrophic forgetting have primarily centered around
developing more effective Continual Learning (CL) methods. In contrast, less
attention was devoted to analyzing the role of network architecture design
(e.g., network depth, width, and components) in contributing to CL. This paper
seeks to bridge this gap between network architecture design and CL, and to
present a holistic study on the impact of network architectures on CL. This
work considers architecture design at the network scaling level, i.e., width
and depth, and also at the network components, i.e., skip connections, global
pooling layers, and down-sampling. In both cases, we first derive insights
through systematically exploring how architectural designs affect CL. Then,
grounded in these insights, we craft a specialized search space for CL and
further propose a simple yet effective ArchCraft method to steer a CL-friendly
architecture, namely, this method recrafts AlexNet/ResNet into AlexAC/ResAC.
Experimental validation across various CL settings and scenarios demonstrates
that improved architectures are parameter-efficient, achieving state-of-the-art
performance of CL while being 86%, 61%, and 97% more compact in terms of
parameters than the naive CL architecture in Class IL and Task IL. Code is
available at this https URL

本文研究网路架构设计对连续学习的影响，以通过网络宽度、深度和组件等方面的系统性研究，提出了一种 CL 友好的网络架构设计方法，该方法在参数方面比传统的 CL 架构紧凑 86%，61% 和 97% 的参数，在 Class IL 和 Task IL 方面实现了最新的连续学习性能。