Recent parameter-efficient language model tuning (PELT) methods manage to
match the performance of fine-tuning with much fewer trainable parameters and
perform especially well when training data is limited. However, different PELT
methods may perform rather differently on the same task, making it nontrivial
to select the most appropriate method for a specific task, especially
considering the fast-growing number of new PELT methods and tasks. In light of
model diversity and the difficulty of model selection, we propose a unified
framework, UniPELT, which incorporates different PELT methods as submodules and
learns to activate the ones that best suit the current data or task setup via
gating mechanism. On the GLUE benchmark, UniPELT consistently achieves 1~4%
gains compared to the best individual PELT method that it incorporates and even
outperforms fine-tuning under different setups. Moreover, UniPELT generally
surpasses the upper bound that takes the best performance of all its submodules
used individually on each task, indicating that a mixture of multiple PELT
methods may be inherently more effective than single methods.

该研究提出了 UniPELT 框架，将不同的 PELT 方法作为子模块，并学习通过门控机制激活最适合当前数据或任务设置的子模块。在 GLUE 基准测试中，UniPELT 相对于其单独包含的最佳 PELT 方法都能够获得 1-4％的收益，并甚至在不同设置下优于 fine-tuning。此外，UniPELT 通常超过单个子模块在每个任务上单独使用的最佳性能上界，表明多个 PELT 方法的混合可能本质上更有效。