Conventional fine-tuning encounters increasing difficulties given the size of
current Pre-trained Language Models, which makes parameter-efficient tuning
become the focal point of frontier research. Previous methods in this field add
tunable adapters into MHA or/and FFN of Transformer blocks to enable PLMs
achieve transferability. However, as an important part of Transformer
architecture, the power of layer normalization for parameter-efficent tuning is
ignored. In this paper, we first propose LN-tuning, by tuning the gain and bias
term of Layer Normalization module with only 0.03\% parameters, which is of
high time-efficency and significantly superior to baselines which are less than
0.1\% tunable parameters. Further, we study the unified framework of combining
LN-tuning with previous ones and we find that: (1) the unified framework of
combining prefix-tuning, the adapter-based method working on MHA, and LN-tuning
achieves SOTA performance. (2) unified framework which tunes MHA and LayerNorm
simultaneously can get performance improvement but those which tune FFN and
LayerNorm simultaneous will cause performance decrease. Ablation study
validates LN-tuning is of no abundant parameters and gives a further
understanding of it.

本文提出了一种仅利用 0.03% 参数的 LN-tuning 方法，通过调整 Layer Normalization 模块的增益和偏置项，实现对 PLMs 的参数高效调整，同时研究了将 LN-tuning 与其他先前方法相结合的统一框架，在 MHA 和 LayerNorm 同时调整的情况下可以获得性能提升，但在 FFN 和 LayerNorm 同时调整的情况下将导致性能下降，该方法实现了 SOTA 的性能表现。