Parameter-Efficient Tuning (PETuning) methods have been deemed by many as the
new paradigm for using pretrained language models (PLMs). By tuning just a
fraction amount of parameters comparing to full model finetuning, PETuning
methods claim to have achieved performance on par with or even better than
finetuning. In this work, we take a step back and re-examine these PETuning
methods by conducting the first comprehensive investigation into the training
and evaluation of them. We found the problematic validation and testing
practice in current studies, when accompanied by the instability nature of
PETuning methods, has led to unreliable conclusions. When being compared under
a truly fair evaluation protocol, PETuning cannot yield consistently
competitive performance while finetuning remains to be the best-performing
method in medium- and high-resource settings. We delve deeper into the cause of
the instability and observed that the number of trainable parameters and
training iterations are two main factors: reducing trainable parameters and
prolonging training iterations may lead to higher stability in PETuning
methods.

本文对参数节省调整方法（PETuning）的训练和评估进行了首次全面调查。研究发现当前 PETuning 研究中存在问题的验证和测试方法，以及 PETuning 方法的不稳定性，导致结论不可靠。当在真正公平的评估协议下进行比较时，PETuning 无法产生一致的竞争性表现，而在中高资源设置中细调仍然是表现最佳的方法。本文深入探讨了 PETuning 方法的不稳定性原因，观察到可训练参数数量和训练迭代次数是两个主要因素：减少可训练参数和延长训练迭代次数可能会导致 PETuning 方法的稳定性更高。