With the growing size of pre-trained models, full fine-tuning and storing all
the parameters for various downstream tasks is costly and infeasible. In this
paper, we propose a new parameter-efficient fine-tuning method, Gradient-based
Parameter Selection (GPS), demonstrating that only tuning a few selected
parameters from the pre-trained model while keeping the remainder of the model
frozen can generate similar or better performance compared with the full model
fine-tuning method. Different from the existing popular and state-of-the-art
parameter-efficient fine-tuning approaches, our method does not introduce any
additional parameters and computational costs during both the training and
inference stages. Another advantage is the model-agnostic and non-destructive
property, which eliminates the need for any other design specific to a
particular model. Compared with the full fine-tuning, GPS achieves 3.33%
(91.78% vs. 88.45%, FGVC) and 9.61% (73.1% vs. 65.57%, VTAB) improvement of the
accuracy with tuning only 0.36% parameters of the pre-trained model on average
over 24 image classification tasks; it also demonstrates a significant
improvement of 17% and 16.8% in mDice and mIoU, respectively, on medical image
segmentation task. Moreover, GPS achieves state-of-the-art performance compared
with existing PEFT methods.

利用渐变参数选择技术（GPS）来进行参数有效的微调，以提高预训练模型的性能，并证明与全模型微调方法相比，只调整预训练模型中的少数参数并保持其余参数冻结可以实现类似或更好的性能，同时不引入额外的参数和计算成本，并在多个任务中表现出更好的性能。