Existing prompt-tuning methods have demonstrated impressive performances in
continual learning (CL), by selecting and updating relevant prompts in the
vision-transformer models. On the contrary, this paper aims to learn each task
by tuning the prompts in the direction orthogonal to the subspace spanned by
previous tasks' features, so as to ensure no interference on tasks that have
been learned to overcome catastrophic forgetting in CL. However, different from
the orthogonal projection in the traditional CNN architecture, the prompt
gradient orthogonal projection in the ViT architecture shows completely
different and greater challenges, i.e., 1) the high-order and non-linear
self-attention operation; 2) the drift of prompt distribution brought by the
LayerNorm in the transformer block. Theoretically, we have finally deduced two
consistency conditions to achieve the prompt gradient orthogonal projection,
which provide a theoretical guarantee of eliminating interference on previously
learned knowledge via the self-attention mechanism in visual prompt tuning. In
practice, an effective null-space-based approximation solution has been
proposed to implement the prompt gradient orthogonal projection. Extensive
experimental results demonstrate the effectiveness of anti-forgetting on four
class-incremental benchmarks with diverse pre-trained baseline models, and our
approach achieves superior performances to state-of-the-art methods. Our code
is available in the supplemental material.

通过对以前任务特征生成的子空间正交方向上调整提示信息，本文提出了在视觉提示调整中实现提示梯度正交投影的一种有效的空间近似解决方案。实验证明了该方法在四个增量学习基准测试中对防止遗忘的有效性，并且优于现有方法的表现。

零空间中的视觉指示调节对于连续学习的影响

Visual Prompt Tuning in Null Space for Continual Learning

Prompt-tuning methods have shown comparable performance as
parameter-efficient fine-tuning (PEFT) methods in various natural language
understanding tasks. However, existing prompt tuning methods still utilize the
entire model architecture; thus, they fail to accelerate inference speed in the
application. In this paper, we propose a novel approach called SKIll-localized
Prompt tuning (SKIP), which is extremely efficient in inference time. Our
method significantly enhances inference efficiency by investigating and
utilizing a skill-localized subnetwork in a language model. Surprisingly, our
method improves the inference speed up to 160% while pruning 52% of the
parameters. Furthermore, we demonstrate that our method is applicable across
various transformer-based architectures, thereby confirming its practicality
and scalability.

提出一种名为 SKIll-localized Prompt tuning (SKIP) 的新方法，能够显著地提高推断效率，同时减少参数数量，适用于各种基于 transformer 的架构。

SKIP: 提高推理速度的技能本地化提示调优

SKIP: Skill-Localized Prompt Tuning for Inference Speed Boost-Up

Prompt-tuning methods for Continual Learning (CL) freeze a large pre-trained
model and focus training on a few parameter vectors termed prompts. Most of
these methods organize these vectors in a pool of key-value pairs, and use the
input image as query to retrieve the prompts (values). However, as keys are
learned while tasks progress, the prompting selection strategy is itself
subject to catastrophic forgetting, an issue often overlooked by existing
approaches. For instance, prompts introduced to accommodate new tasks might end
up interfering with previously learned prompts. To make the selection strategy
more stable, we ask a foundational model (CLIP) to select our prompt within a
two-level adaptation mechanism. Specifically, the first level leverages
standard textual prompts for the CLIP textual encoder, leading to stable class
prototypes. The second level, instead, uses these prototypes along with the
query image as keys to index a second pool. The retrieved prompts serve to
adapt a pre-trained ViT, granting plasticity. In doing so, we also propose a
novel residual mechanism to transfer CLIP semantics to the ViT layers. Through
extensive analysis on established CL benchmarks, we show that our method
significantly outperforms both state-of-the-art CL approaches and the zero-shot
CLIP test. Notably, our findings hold true even for datasets with a substantial
domain gap w.r.t. the pre-training knowledge of the backbone model, as
showcased by experiments on satellite imagery and medical datasets.

通过在 Continual Learning 中引入 Prompt-tuning 方法，以更稳定的选择策略和适应机制，提出了一种能够优于最先进的方法的新模型，即使在与预训练模型存在显著领域差异的数据集上也适用。