Parameter efficient transfer learning (PETL) is an emerging research spot
that aims to adapt large-scale pre-trained models to downstream tasks. Recent
advances have achieved great success in saving storage and computation costs.
However, these methods do not take into account instance-specific visual clues
for visual tasks. In this paper, we propose a Dynamic Visual Prompt Tuning
framework (DVPT), which can generate a dynamic instance-wise token for each
image. In this way, it can capture the unique visual feature of each image,
which can be more suitable for downstream visual tasks. We designed a Meta-Net
module that can generate learnable prompts based on each image, thereby
capturing dynamic instance-wise visual features. Extensive experiments on a
wide range of downstream recognition tasks show that DVPT achieves superior
performance than other PETL methods. More importantly, DVPT even outperforms
full fine-tuning on 17 out of 19 downstream tasks while maintaining high
parameter efficiency. Our code will be released soon.

参数有效的迁移学习 (PETL) 是一个新兴的研究领域，旨在将大规模预训练模型适应下游任务。我们提出了一个动态视觉提示调整框架 (DVPT)，可以为每个图像生成动态的实例级标记，以捕捉每个图像的独特视觉特征，从而更适合下游视觉任务。通过对广泛的下游识别任务进行实验，我们发现 DVPT 方法在性能上优于其他 PETL 方法，甚至在 19 个下游任务中有 17 个超越了完全微调的性能，同时保持了高参数效率。

动态视觉提示调整用于参数高效的迁移学习

Dynamic Visual Prompt Tuning for Parameter Efficient Transfer Learning

Limited labeled data makes it hard to train models from scratch in medical
domain, and an important paradigm is pre-training and then fine-tuning. Large
pre-trained models contain rich representations, which can be adapted to
downstream medical tasks. However, existing methods either tune all the
parameters or the task-specific layers of the pre-trained models, ignoring the
input variations of medical images, and thus they are not efficient or
effective. In this work, we aim to study parameter-efficient fine-tuning (PEFT)
for medical image analysis, and propose a dynamic visual prompt tuning method,
named DVPT. It can extract knowledge beneficial to downstream tasks from large
models with a few trainable parameters. Firstly, the frozen features are
transformed by an lightweight bottleneck layer to learn the domain-specific
distribution of downstream medical tasks, and then a few learnable visual
prompts are used as dynamic queries and then conduct cross-attention with the
transformed features, attempting to acquire sample-specific knowledge that are
suitable for each sample. Finally, the features are projected to original
feature dimension and aggregated with the frozen features. This DVPT module can
be shared between different Transformer layers, further reducing the trainable
parameters. To validate DVPT, we conduct extensive experiments with different
pre-trained models on medical classification and segmentation tasks. We find
such PEFT method can not only efficiently adapt the pre-trained models to the
medical domain, but also brings data efficiency with partial labeled data. For
example, with 0.5\% extra trainable parameters, our method not only outperforms
state-of-the-art PEFT methods, even surpasses the full fine-tuning by more than
2.20\% Kappa score on medical classification task. It can saves up to 60\%
labeled data and 99\% storage cost of ViT-B/16.

通过引入动态视觉提示调整方法，本研究提出了一种适应多样的医学图像输入变化的参数高效微调方法，命名为 DVPT，该方法从大模型中提取有益于下游任务的知识，可通过仅使用少量可训练参数从零开始训练医学图像分析模型。