Prompt-tuning, which freezes pretrained language models (PLMs) and only
fine-tunes few parameters of additional soft prompt, shows competitive
performance against full-parameter fine-tuning (i.e.model-tuning) when the PLM
has billions of parameters, but still performs poorly in the case of smaller
PLMs. Hence, prompt transfer (PoT), which initializes the target prompt with
the trained prompt of similar source tasks, is recently proposed to improve
over prompt-tuning. However, such a vanilla PoT approach usually achieves
sub-optimal performance, as (i) the PoT is sensitive to the similarity of
source-target pair and (ii) directly fine-tuning the prompt initialized with
source prompt on target task might lead to catastrophic forgetting of source
knowledge. In response to these problems, we propose a new metric to accurately
predict the prompt transferability (regarding (i)), and a novel PoT approach
(namely PANDA) that leverages the knowledge distillation technique to transfer
the "knowledge" from the source prompt to the target prompt in a subtle manner
and alleviate the catastrophic forgetting effectively (regarding (ii)).
Furthermore, to achieve adaptive prompt transfer for each source-target pair,
we use our metric to control the knowledge transfer in our PANDA approach.
Extensive and systematic experiments on 189 combinations of 21 source and 9
target datasets across 5 scales of PLMs demonstrate that: 1) our proposed
metric works well to predict the prompt transferability; 2) our PANDA
consistently outperforms the vanilla PoT approach by 2.3% average score (up to
24.1%) among all tasks and model sizes; 3) with our PANDA approach,
prompt-tuning can achieve competitive and even better performance than
model-tuning in various PLM scales scenarios. Code and models will be released
upon acceptance.

该研究提出了一种新的度量方法和 PANDA 方法来解决预训练语言模型中的 prompt 转移问题，PANDA 方法使用知识蒸馏技术来迁移源 prompt 中的知识到目标 prompt 中，从而避免源知识的灾难性遗忘。实验证明，该方法在各种 PLM 尺度场景下的性能均优于传统的 prompt 转移方法和模型调参方法。