Fine-tuning large pre-trained language models on various downstream tasks
with whole parameters is prohibitively expensive. Hence, Parameter-efficient
fine-tuning has attracted attention that only optimizes a few task-specific
parameters with the frozen pre-trained model. In this work, we focus on prefix
tuning, which only optimizes continuous prefix vectors (i.e. pseudo tokens)
inserted into Transformer layers. Based on the observation that the learned
syntax and semantics representation varies a lot at different layers, we argue
that the adaptive prefix will be further tailored to each layer than the fixed
one, enabling the fine-tuning more effective and efficient. Thus, we propose
Adaptive Prefix Tuning (APT) to adjust the prefix in terms of both fine-grained
token level and coarse-grained layer level with a gate mechanism. Experiments
on the SuperGLUE and NER datasets show the effectiveness of APT. In addition,
taking the gate as a probing, we validate the efficiency and effectiveness of
the variable prefix.

本研究聚焦于前缀调整 fine-tuning 技术，并提出了自适应前缀调整（APT）方法，结合门控机制实现对细粒度和粗粒度级别的前缀优化，实证实现了在 SuperGLUE 和 NER 数据集上的有效性和效率验证。