In recent years, prompt tuning has proven effective in adapting pre-trained
vision-language models to downstream tasks. These methods aim to adapt the
pre-trained models by introducing learnable prompts while keeping pre-trained
weights frozen. However, learnable prompts can affect the internal
representation within the self-attention module, which may negatively impact
performance variance and generalization, especially in data-deficient settings.
To address these issues, we propose a novel approach, Read-only Prompt
Optimization (RPO). RPO leverages masked attention to prevent the internal
representation shift in the pre-trained model. Further, to facilitate the
optimization of RPO, the read-only prompts are initialized based on special
tokens of the pre-trained model. Our extensive experiments demonstrate that RPO
outperforms CLIP and CoCoOp in base-to-new generalization and domain
generalization while displaying better robustness. Also, the proposed method
achieves better generalization on extremely data-deficient settings, while
improving parameter efficiency and computational overhead. Code is available at
this https URL

最近几年，迅速调整在适应下游任务的预训练视觉语言模型方面已被证明是有效的。我们提出了一种新颖的方法，只读提示优化 (RPO)，它利用遮罩注意力来防止预训练模型中的内部表示偏移。我们的实验结果表明，RPO 在基于新的泛化和域泛化方面优于 CLIP 和 CoCoOp，同时具有更好的鲁棒性。此外，该方法在极度数据不足的情况下实现了更好的泛化，同时改善了参数效率和计算开销。