Recently, prompt tuning (PT) has gained increasing attention as a
parameter-efficient way of tuning pre-trained language models (PLMs). Despite
extensively reducing the number of tunable parameters and achieving satisfying
performance, PT is training-inefficient due to its slow convergence. To improve
PT's training efficiency, we first make some novel observations about the
prompt transferability of "partial PLMs", which are defined by compressing a
PLM in depth or width. We observe that the soft prompts learned by different
partial PLMs of various sizes are similar in the parameter space, implying that
these soft prompts could potentially be transferred among partial PLMs.
Inspired by these observations, we propose Fast Prompt Tuning (FPT), which
starts by conducting PT using a small-scale partial PLM, and then progressively
expands its depth and width until the full-model size. After each expansion, we
recycle the previously learned soft prompts as initialization for the enlarged
partial PLM and then proceed PT. We demonstrate the feasibility of FPT on 5
tasks and show that FPT could save over 30% training computations while
achieving comparable performance.

本论文提出了一种名为 Fast Prompt Tuning 的技术，通过将 partial PLMs 中的 soft prompts 转化到整个 PLM 中来提高 prompt tuning（PT）的训练效率，该技术的应用可以在保持性能的同时节省 30% 的训练计算资源。