Motivated by the efficiency and rapid convergence of pre-trained models for
solving downstream tasks, this paper extensively studies the impact of
Continual Learning (CL) models as pre-trainers. In both supervised and
unsupervised CL, we find that the transfer quality of the representation often
increases gradually without noticeable degradation in fine-tuning performance.
This is because CL models can learn improved task-general features when easily
forgetting task-specific knowledge. Based on this observation, we suggest a new
unsupervised CL framework with masked modeling, which aims to capture fluent
task-generic representation during training. Furthermore, we propose a new
fine-tuning scheme, GLobal Attention Discretization (GLAD), that preserves rich
task-generic representation during solving downstream tasks. The model
fine-tuned with GLAD achieves competitive performance and can also be used as a
good pre-trained model itself. We believe this paper breaks the barriers
between pre-training and fine-tuning steps and leads to a sustainable learning
framework in which the continual learner incrementally improves model
generalization, yielding better transfer to unseen tasks.

本文的研究重点是探讨 Continual Learning 模型作为预训练器的效果，研究表明，CL 模型可以学习到改进的任务 - 一般特征，在保留丰富任务 - 一般特征的同时，运用 GLobal Attention Discretization fine-tuning 可获得竞争力的性能表现。