Prompt tuning learns soft prompts to condition frozen Pre-trained Language
Models (PLMs) for performing downstream tasks in a parameter-efficient manner.
While prompt tuning has gradually reached the performance level of fine-tuning
as the model scale increases, there is still a large performance gap between
prompt tuning and fine-tuning for models of moderate and small scales
(typically less than 11B parameters). In this paper, we empirically show that
the trained prompt tokens can have a negative impact on a downstream task and
thus degrade its performance. To bridge the gap, we propose a novel Prompt
tuning model with an eXtremely small scale (XPrompt) under the regime of
lottery tickets hypothesis. Specifically, XPrompt eliminates the negative
prompt tokens at different granularity levels through a hierarchical structured
pruning, yielding a more parameter-efficient prompt yet with a competitive
performance. Comprehensive experiments are carried out on SuperGLUE tasks, and
the extensive results indicate that XPrompt is able to close the performance
gap at smaller model scales.

本文提出了一种名为 XPrompt 的新型 Prompt tuning 模型，采用分层结构裁剪方法消除负面的 prompt token 从而优化下游任务的表现，在 SuperGLUE 测试中，在小型模型中能够接近或优于微调方法的性能水平。