Due to increasing interest in adapting models on resource-constrained edges, parameter-efficient transfer learning has been widely explored. Among various methods, Visual Prompt Tuning (VPT), prepending learnable prompts to input space, shows competitive fine-tuning performance compared to training of full network parameters. However, VPT increases the number of input tokens, resulting in additional computational overhead. In this paper, we analyze the impact of the number of prompts on fine-tuning performance and self-attention operation in a vision transformer architecture. Through theoretical and empirical analysis we show that adding more prompts does not lead to linear performance improvement. Further, we propose a Prompt Condensation (PC) technique that aims to prevent performance degradation from using a small number of prompts. We validate our methods on FGVC and VTAB-1k tasks and show that our approach reduces the number of prompts by ~70% while maintaining accuracy.

本文研究了视觉转换器结构中 Prompt 数量对微调效果和自注意力操作的影响。通过理论和实证分析，我们发现增加 Prompt 数量并不能带来线性的性能提升。为此，我们提出 Prompt Condensation 技术来防止 Prompt 数量过多导致的性能下降，实验证明我们的方法在维持准确度的同时能减少大约 70% 的 prompts 数量。

我们真的需要大量的视觉提示吗？