Prompt tuning based on Context Optimization (CoOp) effectively adapts
visual-language models (VLMs) to downstream tasks by inferring additional
learnable prompt tokens. However, these tokens are less discriminative as they
are independent of the pre-trained tokens and fail to capture input-specific
knowledge, such as class-aware textual or instance-aware visual knowledge.
Leveraging the discriminative and generalization capabilities inherent in
pre-trained tokens, we introduce a novel approach named Self-Enhanced Prompt
Tuning (SEP). The core principle of SEP involves adapting the learnable prompt
tokens at each encoder layer from the corresponding self-pretrained tokens,
thereby explicitly incorporating discriminative prior knowledge to enhance both
textual-level and visual-level embeddings. Furthermore, SEP's self-enhanced
tokens not only boost discrimination but also mitigate domain shifts in unseen
domains, enhancing generalization. In practice, SEP selects several
representative tokens from all pre-trained tokens for each input data at every
layer of the text/visual encoders. Subsequently, a Token Fusion Module (TFM) is
introduced to generate a self-enhanced token by merging these representative
tokens with the learnable tokens using a cross-attention mechanism. This
self-enhanced token is then concatenated with all pre-trained tokens, serving
as input for subsequent encoder layers to produce the relevant embeddings.
Comprehensive evaluations across various benchmarks and tasks confirm SEP's
efficacy in prompt tuning. Code: \href{Code}{this https URL}.

基于 Context Optimization (CoOp) 的 Prompt tuning 方式可通过推断附加可学习的提示符令牌来有效地调整视觉 - 语言模型（VLMs）以适应下游任务。然而，这些令牌由于与预训练的令牌无关且无法捕捉特定于输入的知识，如类别感知的文本或实例感知的视觉知识，因此没有足够的区分性。因此，我们提出了一种名为 Self-Enhanced Prompt Tuning (SEP) 的新方法，通过从相应的自预训练的令牌中调整每个编码器层的可学习提示符令牌，从而明确地将区分性先验知识融入文本级别和视觉级别的嵌入中以增强性能。SEP 的自增强令牌不仅增强了区分性，还减轻了未见域中的领域转移，提高了泛化能力。在实践中，SEP 为每个输入数据在文本 / 视觉编码器的每个层级上选择几个代表性令牌。随后，引入了一个 Token Fusion Module (TFM)，通过使用交叉注意机制将这些代表性令牌与可学习令牌合并，生成一个自增强令牌。这个自增强令牌随后与所有预训练令牌连接，作为后续编码器层的输入，生成相关的嵌入。通过在各种基准和任务上进行全面评估，证实了 SEP 在提示调整中的有效性。