Exploiting the power of pre-trained models, prompt-based approaches stand out
compared to other continual learning solutions in effectively preventing
catastrophic forgetting, even with very few learnable parameters and without
the need for a memory buffer. While existing prompt-based continual learning
methods excel in leveraging prompts for state-of-the-art performance, they
often lack a theoretical explanation for the effectiveness of prompting. This
paper conducts a theoretical analysis to unravel how prompts bestow such
advantages in continual learning, thus offering a new perspective on prompt
design. We first show that the attention block of pre-trained models like
Vision Transformers inherently encodes a special mixture of experts
architecture, characterized by linear experts and quadratic gating score
functions. This realization drives us to provide a novel view on prefix tuning,
reframing it as the addition of new task-specific experts, thereby inspiring
the design of a novel gating mechanism termed Non-linear Residual Gates
(NoRGa). Through the incorporation of non-linear activation and residual
connection, NoRGa enhances continual learning performance while preserving
parameter efficiency. The effectiveness of NoRGa is substantiated both
theoretically and empirically across diverse benchmarks and pretraining
paradigms.

基于预训练模型的提示式方法在连续学习中与其他解决方案相比，在仅有很少可学习参数且无需内存缓冲区的情况下，展现出了防止灾难性遗忘的强大能力。本研究在理论上分析和解释了提示式学习的有效性，并提出了一种基于非线性残差门机制（NoRGa）的新型门机制，从而在保持参数效率的同时提高了连续学习性能。