Parameter-efficient fine-tuning for pre-trained Vision Transformers aims to
adeptly tailor a model to downstream tasks by learning a minimal set of new
adaptation parameters while preserving the frozen majority of pre-trained
parameters. Striking a balance between retaining the generalizable
representation capacity of the pre-trained model and acquiring task-specific
features poses a key challenge. Currently, there is a lack of focus on guiding
this delicate trade-off. In this study, we approach the problem from the
perspective of Singular Value Decomposition (SVD) of pre-trained parameter
matrices, providing insights into the tuning dynamics of existing methods.
Building upon this understanding, we propose a Residual-based Low-Rank
Rescaling (RLRR) fine-tuning strategy. This strategy not only enhances
flexibility in parameter tuning but also ensures that new parameters do not
deviate excessively from the pre-trained model through a residual design.
Extensive experiments demonstrate that our method achieves competitive
performance across various downstream image classification tasks, all while
maintaining comparable new parameters. We believe this work takes a step
forward in offering a unified perspective for interpreting existing methods and
serves as motivation for the development of new approaches that move closer to
effectively considering the crucial trade-off mentioned above. Our code is
available at
\href{https://github.com/zstarN70/RLRR.git}{this https URL}.

利用奇异值分解对预训练参数矩阵进行精细调整的高效参数微调方法提供了对现有方法调整动态的洞察，通过残差设计确保新参数不会过度偏离预训练模型，实验结果表明，这种方法在各种下游图像分类任务中实现了竞争性的性能，并且保持了可比较的新参数。