Prompt tuning is a parameter-efficient tuning (PETuning) method for utilizing
pre-trained models (PTMs) that simply prepends a soft prompt to the input and
only optimizes the prompt to adapt PTMs to downstream tasks. Although it is
parameter- and deployment-efficient, its performance still lags behind other
state-of-the-art PETuning methods. Besides, the training cost of prompt tuning
is not significantly reduced due to the back-propagation through the entire
model. Through empirical analyses, we shed some light on the lagging
performance of prompt tuning and recognize a trade-off between the propagation
distance from label signals to the inserted prompt and the influence of the
prompt on model outputs. Further, we present Late Prompt Tuning (LPT) that
inserts a late prompt into an intermediate layer of the PTM instead of the
input layer or all layers. The late prompt is obtained by a neural prompt
generator conditioned on the hidden states before the prompt insertion layer
and therefore is instance-dependent. Through extensive experimental results
across various tasks and PTMs, we show that LPT can achieve competitive
performance to full model tuning and other PETuning methods under both
full-data and few-shot scenarios while possessing faster training speed and
lower memory cost.

该研究提出了一种名为 Late Prompt Tuning (LPT) 的 PETuning 方法，它将追加的提示插入到 PTM 的中间层而非输入层或所有层，并通过一个神经提示生成器获得实例依赖的提示，具有更快的训练速度和更低的内存成本，可在全数据和少样本场景下实现与全模型调整和其他 PETuning 方法竞争的性能。