Vision-language models (VLMs), such as CLIP, have demonstrated impressive
zero-shot capabilities for various downstream tasks. Their performance can be
further enhanced through few-shot prompt tuning methods. However, current
studies evaluate the performance of learned prompts separately on base and new
classes. This evaluation lacks practicality for real-world applications since
downstream tasks cannot determine whether the data belongs to base or new
classes in advance. In this paper, we explore a problem setting called
Open-world Prompt Tuning (OPT), which involves tuning prompts on base classes
and evaluating on a combination of base and new classes. By introducing
Decomposed Prompt Tuning framework (DePT), we theoretically demonstrate that
OPT can be solved by incorporating out-of-distribution detection into prompt
tuning, thereby enhancing the base-to-new discriminability. Based on DePT, we
present a novel prompt tuning approach, namely, Decomposed Context Optimization
(DeCoOp), which introduces new-class detectors and sub-classifiers to further
enhance the base-class and new-class discriminability. Experimental results on
11 benchmark datasets validate the effectiveness of DePT and demonstrate that
DeCoOp outperforms current state-of-the-art methods, providing a significant 2%
average accuracy improvement.

该研究论文介绍了一种名为 Open-world Prompt Tuning (OPT) 的问题设置，通过引入 Decomposed Prompt Tuning framework (DePT) 解决 OPT 问题，并提出了一种名为 Decomposed Context Optimization (DeCoOp) 的新型 prompt tuning 方法，实验证明 DePT 的有效性，并显示 DeCoOp 相较于当前最先进的方法有着 2% 的平均准确率的提升。

DeCoOp：具有 OD 检测的鲁棒提示调节

DeCoOp: Robust Prompt Tuning with Out-of-Distribution Detection

Prompt tuning is one of the most effective solutions to adapting a fixed
pre-trained language model (PLM) for various downstream tasks, especially with
only a few input samples. However, the security issues, e.g., Trojan attacks,
of prompt tuning on a few data samples are not well-studied. Transferring
established data poisoning attacks directly to few-shot prompt tuning presents
multiple challenges. One significant issue is the \textit{poisoned imbalance
issue}, where non-target class samples are added to the target class, resulting
in a greater number of target-class samples compared to non-target class. While
this issue is not critical in regular tuning, it significantly hampers the
few-shot prompt tuning, making it difficult to simultaneously achieve a high
attack success rate (ASR) and maintain clean data accuracy (CDA). Additionally,
few-shot prompting is prone to overfitting in terms of both ASR and CDA. In
this paper, we introduce \textit{TrojFSP}, a method designed to address the
challenges. To solve the poisoned imbalance issue, we develop a
\textit{Target-Class Shrink (TC-Shrink)} technique, which aims to equalize the
number of poisoning samples. To combat overfitting, we employ a
\textit{Selective Token Poisoning} technique to boost attack performance.
Furthermore, we introduce a \textit{Trojan-Trigger Attention} objective
function to amplify the attention of the poisoned trojan prompt on triggers.
Experiments show that our TrojFSP achieves an ASR of over 99\% while
maintaining negligible decreases in CDA across various PLMs and datasets.

本文介绍了一种名为 TrojFSP 的方法，用于解决少样本快速提示调整中的安全问题和过拟合问题，通过目标类样本缩减技术、有选择性的令牌污染技术和特洛伊触发器注意目标函数，实现了在各种预训练语言模型和数据集上 ASR 超过 99％，对 CDA 的减少可以忽略。

TrojFSP：在少样本提示调整中插入特洛伊

TrojFSP: Trojan Insertion in Few-shot Prompt Tuning

Vision-language models such as CLIP learn a generic text-image embedding from
large-scale training data. A vision-language model can be adapted to a new
classification task through few-shot prompt tuning. We find that such a prompt
tuning process is highly robust to label noises. This intrigues us to study the
key reasons contributing to the robustness of the prompt tuning paradigm. We
conducted extensive experiments to explore this property and find the key
factors are: 1) the fixed classname tokens provide a strong regularization to
the optimization of the model, reducing gradients induced by the noisy samples;
2) the powerful pre-trained image-text embedding that is learned from diverse
and generic web data provides strong prior knowledge for image classification.
Further, we demonstrate that noisy zero-shot predictions from CLIP can be used
to tune its own prompt, significantly enhancing prediction accuracy in the
unsupervised setting. The code is available at this https URL

视觉语言模型通过大规模训练数据学习了一种通用的文本 - 图像嵌入。通过少量样本的提示调整，可以将视觉语言模型适应到新的分类任务中。我们发现，这种提示调整过程对标签噪声非常稳健，且具有鲁棒性。