Parameter-efficient tuning has become a trend in transferring large-scale
foundation models to downstream applications. Existing methods typically embed
some light-weight tuners into the backbone, where both the design and the
learning of the tuners are highly dependent on the base model. This work offers
a new tuning paradigm, dubbed Res-Tuning, which intentionally unbinds tuners
from the backbone. With both theoretical and empirical evidence, we show that
popular tuning approaches have their equivalent counterparts under our
unbinding formulation, and hence can be integrated into our framework
effortlessly. Thanks to the structural disentanglement, we manage to free the
design of tuners from the network architecture, facilitating flexible
combination of various tuning strategies. We further propose a memory-efficient
variant of Res-Tuning, where the bypass i.e., formed by a sequence of tuners)
is effectively detached from the main branch, such that the gradients are
back-propagated only to the tuners but not to the backbone. Such a detachment
also allows one-time backbone forward for multi-task inference. Extensive
experiments on both discriminative and generative tasks demonstrate the
superiority of our method over existing alternatives from the perspectives of
efficacy and efficiency. Project page:
$\href{https://res-tuning.github.io/}{\textit{this https URL}}$.

参数高效调优的新范式 Res-Tuning 将调优器与网络解耦，从而实现多样调优策略的灵活组合，并通过实验证明其在效能和效率方面优于现有替代方案。