Large language models have shown their ability to become effective few-shot
learners with prompting, revoluting the paradigm of learning with data
scarcity. However, this approach largely depends on the quality of prompt
initialization, and always exhibits large variability among different runs.
Such property makes prompt tuning highly unreliable and vulnerable to poorly
constructed prompts, which limits its extension to more real-world
applications. To tackle this issue, we propose to treat the hard prompt and
soft prompt as separate inputs to mitigate noise brought by the prompt
initialization. Furthermore, we optimize soft prompts with contrastive learning
for utilizing class-aware information in the training process to maintain model
performance. Experimental results demonstrate that \sysname outperforms
state-of-the-art methods by 7.20% in accuracy and reduces the standard
deviation by 2.02 on average. Furthermore, extensive experiments underscore its
robustness and stability across 7 datasets covering various tasks.

通过将硬提示和软提示作为单独的输入来解决 prompt 初始化带来的噪音问题，并通过对比学习优化软提示以利用类别感知信息，实验证明该方法在准确性上超过了先进方法 7.20％，平均标准差减少了 2.02，并在涵盖各种任务的 7 个数据集上得到了强大和稳定的结果。

稳定 PT：通过输入分离实现少样本学习的稳定提示

StablePT: Towards Stable Prompting for Few-shot Learning via Input  Separation

Visual prompt tuning (VPT) is a promising solution incorporating learnable
prompt tokens to customize pre-trained models for downstream tasks. However,
VPT and its variants often encounter challenges like prompt initialization,
prompt length, and subpar performance in self-supervised pretraining, hindering
successful contextual adaptation. This study commences by exploring the
correlation evolvement between prompts and patch tokens during proficient
training. Inspired by the observation that the prompt tokens tend to share high
mutual information with patch tokens, we propose initializing prompts with
downstream token prototypes. The strategic initialization, a stand-in for the
previous initialization, substantially improves performance in fine-tuning. To
refine further, we optimize token construction with a streamlined pipeline that
maintains excellent performance with almost no increase in computational
expenses compared to VPT. Exhaustive experiments show our proposed approach
outperforms existing methods by a remarkable margin. For instance, it surpasses
full fine-tuning in 19 out of 24 tasks, using less than 0.4% of learnable
parameters on the FGVC and VTAB-1K benchmarks. Notably, our method
significantly advances the adaptation for self-supervised pretraining,
achieving impressive task performance gains of at least 10% to 30%. Besides,
the experimental results demonstrate the proposed SPT is robust to prompt
lengths and scales well with model capacity and training data size. We finally
provide an insightful exploration into the amount of target data facilitating
the adaptation of pre-trained models to downstream tasks.

通过关键参数的优化和初始化，我们提出了一种改进的视觉提示调整方法，用于优化预训练模型在下游任务中的性能表现，并在大量实验证明该方法在自我监督预训练适应性上取得了显著的性能改进。