Prompt learning has been designed as an alternative to fine-tuning for
adapting Vision-language (V-L) models to the downstream tasks. Previous works
mainly focus on text prompt while visual prompt works are limited for V-L
models. The existing visual prompt methods endure either mediocre performance
or unstable training process, indicating the difficulty of visual prompt
learning. In this paper, we propose a new Progressive Visual Prompt (ProVP)
structure to strengthen the interactions among prompts of different layers.
More importantly, our ProVP could effectively propagate the image embeddings to
deep layers and behave partially similar to an instance adaptive prompt method.
To alleviate generalization deterioration, we further propose a new contrastive
feature re-formation, which prevents the serious deviation of the prompted
visual feature from the fixed CLIP visual feature distribution. Combining both,
our method (ProVP-Ref) is evaluated on 11 image benchmark datasets and achieves
7/11 state-of-theart results on both few-shot and base-to-novel settings. To
the best of our knowledge, we are the first to demonstrate the superior
performance of visual prompts in V-L models to previous prompt-based methods in
downstream tasks. Meanwhile, it implies that our ProVP-Ref shows the best
capability to adapt and to generalize.

本文针对视觉 - 语言模型中的文本提示有限，现有视觉提示方法性能或训练过程不稳定的问题，提出了一种新的渐进式视觉提示结构（ProVP），并结合对比特征重构，最终获得了在 11 个基准数据集上的最佳性能。