Full fine-tuning is a popular approach to adapt Transformer-based pre-trained
large language models to a specific downstream task. However, the substantial
requirements for computational power and storage have discouraged its
widespread use. Moreover, increasing evidence of catastrophic forgetting and
overparameterization in the Transformer architecture has motivated researchers
to seek more efficient fine-tuning (PEFT) methods. Commonly known
parameter-efficient fine-tuning methods like LoRA and BitFit are typically
applied across all layers of the model. We propose a PEFT method, called
Stratified Progressive Adaptation Fine-tuning (SPAFIT), based on the
localization of different types of linguistic knowledge to specific layers of
the model. Our experiments, conducted on nine tasks from the GLUE benchmark,
show that our proposed SPAFIT method outperforms other PEFT methods while
fine-tuning only a fraction of the parameters adjusted by other methods.

本研究提出一种基于模型不同层次具有语言知识定位的 PEFT 方法，称为 Stratified Progressive Adaptation Fine-tuning（SPAFIT）。我们在 GLUE 基准测试中进行的实验证明，我们的 SPAFIT 方法在仅微调其他方法调整参数的一小部分情况下，优于其他 PEFT 方法。