Parameter efficient transfer learning (PETL) is an emerging research spot
that aims to adapt large-scale pre-trained models to downstream tasks. Recent
advances have achieved great success in saving storage and computation costs.
However, these methods do not take into account instance-specific visual clues
for visual tasks. In this paper, we propose a Dynamic Visual Prompt Tuning
framework (DVPT), which can generate a dynamic instance-wise token for each
image. In this way, it can capture the unique visual feature of each image,
which can be more suitable for downstream visual tasks. We designed a Meta-Net
module that can generate learnable prompts based on each image, thereby
capturing dynamic instance-wise visual features. Extensive experiments on a
wide range of downstream recognition tasks show that DVPT achieves superior
performance than other PETL methods. More importantly, DVPT even outperforms
full fine-tuning on 17 out of 19 downstream tasks while maintaining high
parameter efficiency. Our code will be released soon.

参数有效的迁移学习 (PETL) 是一个新兴的研究领域，旨在将大规模预训练模型适应下游任务。我们提出了一个动态视觉提示调整框架 (DVPT)，可以为每个图像生成动态的实例级标记，以捕捉每个图像的独特视觉特征，从而更适合下游视觉任务。通过对广泛的下游识别任务进行实验，我们发现 DVPT 方法在性能上优于其他 PETL 方法，甚至在 19 个下游任务中有 17 个超越了完全微调的性能，同时保持了高参数效率。

动态视觉提示调整用于参数高效的迁移学习

Dynamic Visual Prompt Tuning for Parameter Efficient Transfer Learning

Recently, large-scale pre-trained vision-language models (e.g. CLIP and
ALIGN) have demonstrated remarkable effectiveness in acquiring transferable
visual representations. To leverage the valuable knowledge encoded within these
models for downstream tasks, several fine-tuning approaches, including prompt
tuning methods and adapter-based methods, have been developed to adapt
vision-language models effectively with supervision. However, these methods
rely on the availability of annotated samples, which can be labor-intensive and
time-consuming to acquire, thus limiting scalability. To address this issue, in
this work, we design an unsupervised fine-tuning approach for vision-language
models called Unsupervised Prototype Adapter (UP-Adapter). Specifically, for
the unannotated target datasets, we leverage the text-image aligning capability
of CLIP to automatically select the most confident samples for each class.
Utilizing these selected samples, we generate class prototypes, which serve as
the initialization for the learnable prototype model. After fine-tuning, the
prototype model prediction is combined with the original CLIP's prediction by a
residual connection to perform downstream recognition tasks. Our extensive
experimental results on image recognition and domain generalization show that
the proposed unsupervised method outperforms 8-shot CoOp, 8-shot Tip-Adapter,
and also the state-of-the-art UPL method by large margins.

我们设计了一种名为 Unsupervised Prototype Adapter (UP-Adapter) 的无监督微调方法，通过利用 CLIP 的文本 - 图像对齐能力自动选择每个类别中最有信心的样本，并利用这些选择的样本生成类别原型，用于可学习的原型模型的初始化。经过微调后，通过剩余连接将原型模型的预测与原始 CLIP 的预测相结合，用于执行下游识别任务。我们在图像识别和领域泛化方面的大量实验结果表明，所提出的无监督方法在 8-shot CoOp、8-shot Tip-Adapter 以及最先进的 UPL 方法上都取得了显著优势。

无监督视觉语言模型的原型适配器

Unsupervised Prototype Adapter for Vision-Language Models

We present prompt distribution learning for effectively adapting a
pre-trained vision-language model to address downstream recognition tasks. Our
method not only learns low-bias prompts from a few samples but also captures
the distribution of diverse prompts to handle the varying visual
representations. In this way, we provide high-quality task-related content for
facilitating recognition. This prompt distribution learning is realized by an
efficient approach that learns the output embeddings of prompts instead of the
input embeddings. Thus, we can employ a Gaussian distribution to model them
effectively and derive a surrogate loss for efficient training. Extensive
experiments on 12 datasets demonstrate that our method consistently and
significantly outperforms existing methods. For example, with 1 sample per
category, it relatively improves the average result by 9.1% compared to
human-crafted prompts.

提出了基于提示分布学习的方法，从而通过有效地适应先前训练的视觉 - 语言模型来处理下游的识别任务，该方法不仅从少量样本中学习低偏差的提示，而且还捕获多种提示的分布以处理不同的视觉表示。

分布式学习

Prompt Distribution Learning

The current modus operandi in adapting pre-trained models involves updating
all the backbone parameters, ie, full fine-tuning. This paper introduces Visual
Prompt Tuning (VPT) as an efficient and effective alternative to full
fine-tuning for large-scale Transformer models in vision. Taking inspiration
from recent advances in efficiently tuning large language models, VPT
introduces only a small amount (less than 1% of model parameters) of trainable
parameters in the input space while keeping the model backbone frozen. Via
extensive experiments on a wide variety of downstream recognition tasks, we
show that VPT achieves significant performance gains compared to other
parameter efficient tuning protocols. Most importantly, VPT even outperforms
full fine-tuning in many cases across model capacities and training data
scales, while reducing per-task storage cost.

该论文提出了一种名为 Visual Prompt Tuning（VPT）的高效且有效的调整大规模 Transformer 模型的替代方案，相较于 fine-tuning，VPT 仅在输入空间中引入很少的可训练参数，通过在广泛的下游识别任务上的实验，我们发现 VPT 在许多情况下甚至比全尺寸 fine-tuning 更加优秀，同时减小了每个任务的存储成本。