Prompt-based Continual Learning (PCL) has gained considerable attention as a
promising continual learning solution as it achieves state-of-the-art
performance while preventing privacy violation and memory overhead issues.
Nonetheless, existing PCL approaches face significant computational burdens
because of two Vision Transformer (ViT) feed-forward stages; one is for the
query ViT that generates a prompt query to select prompts inside a prompt pool;
the other one is a backbone ViT that mixes information between selected prompts
and image tokens. To address this, we introduce a one-stage PCL framework by
directly using the intermediate layer's token embedding as a prompt query. This
design removes the need for an additional feed-forward stage for query ViT,
resulting in ~50% computational cost reduction for both training and inference
with marginal accuracy drop < 1%. We further introduce a Query-Pool
Regularization (QR) loss that regulates the relationship between the prompt
query and the prompt pool to improve representation power. The QR loss is only
applied during training time, so there is no computational overhead at
inference from the QR loss. With the QR loss, our approach maintains ~ 50%
computational cost reduction during inference as well as outperforms the prior
two-stage PCL methods by ~1.4% on public class-incremental continual learning
benchmarks including CIFAR-100, ImageNet-R, and DomainNet.

通过引入一种单阶段的 PCL 框架，将中间层的标记嵌入作为提示查询，消除了查询 ViT 的额外前馈阶段，从而在训练和推理中将计算成本降低了约 50%，准确度仅下降不到 1%。此外，引入了查询池正则化损失（QR 损失），用于改进提示查询和提示池之间的关系，该损失仅在训练时应用，因此在推理阶段没有计算开销。通过引入 QR 损失，我们的方法在推理过程中仍然保持了约 50% 的计算成本降低，并且在包括 CIFAR-100、ImageNet-R 和 DomainNet 在内的公共类增量连续学习基准测试中优于之前的两阶段 PCL 方法约 1.4%。

一阶段基于提示的持续学习

One-stage Prompt-based Continual Learning

Prompt-based continual learning is an emerging direction in leveraging
pre-trained knowledge for downstream continual learning, and has almost reached
the performance pinnacle under supervised pre-training. However, our empirical
research reveals that the current strategies fall short of their full potential
under the more realistic self-supervised pre-training, which is essential for
handling vast quantities of unlabeled data in practice. This is largely due to
the difficulty of task-specific knowledge being incorporated into instructed
representations via prompt parameters and predicted by uninstructed
representations at test time. To overcome the exposed sub-optimality, we
conduct a theoretical analysis of the continual learning objective in the
context of pre-training, and decompose it into hierarchical components:
within-task prediction, task-identity inference, and task-adaptive prediction.
Following these empirical and theoretical insights, we propose Hierarchical
Decomposition (HiDe-)Prompt, an innovative approach that explicitly optimizes
the hierarchical components with an ensemble of task-specific prompts and
statistics of both uninstructed and instructed representations, further with
the coordination of a contrastive regularization strategy. Our extensive
experiments demonstrate the superior performance of HiDe-Prompt and its
robustness to pre-training paradigms in continual learning (e.g., up to 15.01%
and 9.61% lead on Split CIFAR-100 and Split ImageNet-R, respectively). Our code
is available at https://github.com/thu-ml/HiDe-Prompt.

通过在自我监督预训练中应用分层分解 (HiDe-Prompt) 的创新方法，通过一系列特定任务的提示和无指导和有指导表示的统计数据的协调，以及对比正则化策略，我们展示了 HiDe-Prompt 在持续学习中的优越性能和对预训练范式的鲁棒性。