Recent parameter-efficient finetuning (PEFT) techniques aim to improve over the considerable cost of fully finetuning large pretrained language models (PLM). As different PEFT techniques proliferate, it is becoming difficult to compare them, in particular in terms of (i) the structure and functionality they add to the PLM, (ii) the different types and degrees of efficiency improvements achieved, (iii) performance at different downstream tasks, and (iv) how differences in structure and functionality relate to efficiency and task performance. To facilitate such comparisons, this paper presents a reference framework which standardises aspects shared by different PEFT techniques, while isolating differences to specific locations and interactions with the standard components. Through this process of standardising and isolating differences, a modular view of PEFT techniques emerges, supporting not only direct comparison of different techniques and their efficiency and task performance, but also systematic exploration of reusability and composability of the different types of finetuned modules. We demonstrate how the reference framework can be applied to understand properties and relative advantages of PEFT techniques, hence to inform selection of techniques for specific tasks, and design choices for new PEFT techniques.

本文提出了一个参考框架，该框架标准化了不同PEFT技术共享的方面，同时将差异隔离到特定位置和与标准组件的交互。通过标准化和隔离差异的过程，PEFT技术的模块化视图出现，既支持不同技术及其效率和任务表现的直接比较，也支持不同类型的调优模块的可重用性和组合性的系统探索。我们演示了如何应用参考框架来了解PEFT技术的属性和相对优势，因此，可以为特定任务选择技术，并为新的PEFT技术设计选择。

PEFT-Ref: 一种用于参数高效微调技术的模块化参考架构和分类方法