Due to increasing interest in adapting models on resource-constrained edges,
parameter-efficient transfer learning has been widely explored. Among various
methods, Visual Prompt Tuning (VPT), prepending learnable prompts to input
space, shows competitive fine-tuning performance compared to training of full
network parameters. However, VPT increases the number of input tokens,
resulting in additional computational overhead. In this paper, we analyze the
impact of the number of prompts on fine-tuning performance and self-attention
operation in a vision transformer architecture. Through theoretical and
empirical analysis we show that adding more prompts does not lead to linear
performance improvement. Further, we propose a Prompt Condensation (PC)
technique that aims to prevent performance degradation from using a small
number of prompts. We validate our methods on FGVC and VTAB-1k tasks and show
that our approach reduces the number of prompts by ~70% while maintaining
accuracy.

本文研究了视觉转换器结构中 Prompt 数量对微调效果和自注意力操作的影响。通过理论和实证分析，我们发现增加 Prompt 数量并不能带来线性的性能提升。为此，我们提出 Prompt Condensation 技术来防止 Prompt 数量过多导致的性能下降，实验证明我们的方法在维持准确度的同时能减少大约 70% 的 prompts 数量。