Parameter-efficient fine-tuning (PEFT) of pre-trained language models has
recently demonstrated remarkable achievements, effectively matching the
performance of full fine-tuning while utilizing significantly fewer trainable
parameters, and consequently addressing the storage and communication
constraints. Nonetheless, various PEFT methods are limited by their inherent
characteristics. In the case of sparse fine-tuning, which involves modifying
only a small subset of the existing parameters, the selection of fine-tuned
parameters is task- and domain-specific, making it unsuitable for federated
learning. On the other hand, PEFT methods with adding new parameters typically
introduce additional inference latency. In this paper, we demonstrate the
feasibility of generating a sparse mask in a task-agnostic manner, wherein all
downstream tasks share a common mask. Our approach, which relies solely on the
magnitude information of pre-trained parameters, surpasses existing
methodologies by a significant margin when evaluated on the GLUE benchmark.
Additionally, we introduce a novel adapter technique that directly applies the
adapter to pre-trained parameters instead of the hidden representation, thereby
achieving identical inference speed to that of full fine-tuning. Through
extensive experiments, our proposed method attains a new state-of-the-art
outcome in terms of both performance and storage efficiency, storing only 0.03%
parameters of full fine-tuning.

本文提出了一种任务不可知的生成稀疏掩码的方法，仅使用预训练参数的振幅信息，可以显著提高性能和存储效率，并引入了一种新颖的适配器技术，可以直接应用于预训练参数，与全细调速度相同。