Existing fine-tuning methods either tune all parameters of the pre-trained
model (full fine-tuning), which is not efficient, or only tune the last linear
layer (linear probing), which suffers a significant accuracy drop compared to
the full fine-tuning. In this paper, we propose a new parameter-efficient
fine-tuning method termed as SSF, representing that researchers only need to
Scale and Shift the deep Features extracted by a pre-trained model to catch up
with the performance of full fine-tuning. In this way, SSF also surprisingly
outperforms other parameter-efficient fine-tuning approaches even with a
smaller number of tunable parameters. Furthermore, different from some existing
parameter-efficient fine-tuning methods (e.g., Adapter or VPT) that introduce
the extra parameters and computational cost in the training and inference
stages, SSF only adds learnable parameters during the training stage, and these
additional parameters can be merged into the original pre-trained model weights
via re-parameterization in the inference phase. With the proposed SSF, our
model obtains 2.46% (90.72% vs. 88.54%) and 11.48% (73.10% vs. 65.57%)
performance improvement on FGVC and VTAB-1k in terms of Top-1 accuracy compared
to the full fine-tuning but only fine-tuning about 0.3M parameters. We also
conduct amounts of experiments in various model families (CNNs, Transformers,
and MLPs) and datasets. Results on 26 image classification datasets in total
and 3 robustness & out-of-distribution datasets show the effectiveness of SSF.
Code is available at this https URL

提出了一种新的参数高效的调优方法（称为 SSF），仅需要缩放和移位预训练模型提取的深度特征即可达到与完全调优相同的性能。该方法仅在训练阶段添加可学习参数，并可以通过重新参数化将这些附加参数合并到原始预训练模型权重中，相比于完全调优表现出更好的性能，特别是对于少量的可调参数。在各种模型家族（CNN、Transformer 和 MLP）和数据集上进行的大量实验表明，SSF 的有效性。