Visual Prompt Learning (VPL) differs from traditional fine-tuning methods in
reducing significant resource consumption by avoiding updating pre-trained
model parameters. Instead, it focuses on learning an input perturbation, a
visual prompt, added to downstream task data for making predictions. Since
learning generalizable prompts requires expert design and creation, which is
technically demanding and time-consuming in the optimization process,
developers of Visual Prompts as a Service (VPaaS) have emerged. These
developers profit by providing well-crafted prompts to authorized customers.
However, a significant drawback is that prompts can be easily copied and
redistributed, threatening the intellectual property of VPaaS developers.
Hence, there is an urgent need for technology to protect the rights of VPaaS
developers. To this end, we present a method named \textbf{WVPrompt} that
employs visual prompt watermarking in a black-box way. WVPrompt consists of two
parts: prompt watermarking and prompt verification. Specifically, it utilizes a
poison-only backdoor attack method to embed a watermark into the prompt and
then employs a hypothesis-testing approach for remote verification of prompt
ownership. Extensive experiments have been conducted on three well-known
benchmark datasets using three popular pre-trained models: RN50, BIT-M, and
Instagram. The experimental results demonstrate that WVPrompt is efficient,
harmless, and robust to various adversarial operations.

通过视觉提示水印技术，在保护 VPaaS 开发者权益的同时，有效提高了效率、防止攻击，并在多种敌对操作下表现出鲁棒性。

保护 VPaaS 中视觉提示的著作权：通过水印防止抄袭

Are You Copying My Prompt? Protecting the Copyright of Vision Prompt for  VPaaS via Watermark

Fine-tuning large pre-trained computer vision models is infeasible for
resource-limited users. Visual prompt learning (VPL) has thus emerged to
provide an efficient and flexible alternative to model fine-tuning through
Visual Prompt as a Service (VPPTaaS). Specifically, the VPPTaaS provider
optimizes a visual prompt given downstream data, and downstream users can use
this prompt together with the large pre-trained model for prediction. However,
this new learning paradigm may also pose security risks when the VPPTaaS
provider instead provides a malicious visual prompt. In this paper, we take the
first step to explore such risks through the lens of backdoor attacks.
Specifically, we propose BadVisualPrompt, a simple yet effective backdoor
attack against VPL. For example, poisoning $5\%$ CIFAR10 training data leads to
above $99\%$ attack success rates with only negligible model accuracy drop by
$1.5\%$. In particular, we identify and then address a new technical challenge
related to interactions between the backdoor trigger and visual prompt, which
does not exist in conventional, model-level backdoors. Moreover, we provide
in-depth analyses of seven backdoor defenses from model, prompt, and input
levels. Overall, all these defenses are either ineffective or impractical to
mitigate our BadVisualPrompt, implying the critical vulnerability of VPL.

通过对视觉提示学习的后门攻击 (BadVisualPrompt)，我们发现对模型、提示和输入等级的七种后门防御都要么无效，要么不切合实际，从而表明了视觉提示学习 (VPL) 的关键性漏洞。

视觉提示学习中的后门

Prompt Backdoors in Visual Prompt Learning

With the introduction of the transformer architecture in computer vision,
increasing model scale has been demonstrated as a clear path to achieving
performance and robustness gains. However, with model parameter counts reaching
the billions, classical finetuning approaches are becoming increasingly
limiting and even unfeasible when models become hosted as inference APIs, as in
NLP. To this end, visual prompt learning, whereby a model is adapted by
learning additional inputs, has emerged as a potential solution for adapting
frozen and cloud-hosted models: During inference, this neither requires access
to the internals of models' forward pass function, nor requires any
post-processing. In this work, we propose the Prompt Generation Network (PGN)
that generates high performing, input-dependent prompts by sampling from an
end-to-end learned library of tokens. We further introduce the "prompt
inversion" trick, with which PGNs can be efficiently trained in a latent space
but deployed as strictly input-only prompts for inference. We show the PGN is
effective in adapting pre-trained models to various new datasets: It surpasses
previous methods by a large margin on 12/12 datasets and even outperforms
full-finetuning on 5/12, while requiring 100x less parameters.

本论文介绍了一种基于视觉提示学习的模型适应方法，提出了 Prompt Generation Network (PGN) 网络，通过从端到端学习的词汇表中采样生成高性能、与输入相关的提示，从而使得预先训练的模型更适应不同的数据集。实验表明，PGN 在 12 个数据集上均优于以前的方法，并且在 5 个数据集上甚至优于全微调，而只需要 100 倍以下的参数。