Res-Tuning introduces a flexible and efficient paradigm for model tuning,
showing that tuners decoupled from the backbone network can achieve performance
comparable to traditional methods. Existing methods commonly construct the
tuner as a set of trainable low-rank decomposition matrices, positing that a
low-rank subspace suffices for adapting pre-trained foundational models to new
scenarios. In this work, we present an advanced, efficient tuner augmented with
low-rank attention, termed Res-Attn , which also adheres to the Res-Tuning
framework. Res-Attn utilizes a parallel multi-head attention module equipped
with low-rank projections for query, key, and value to execute streamlined
attention operations. Through training this lightweight attention module,
Res-Attn facilitates adaptation to new scenarios. Our extensive experiments
across a range of discriminative and generative tasks showcase the superior
performance of our method when compared to existing alternatives

基于 Res-Tuning 框架，本研究提出了一种高级高效的调谐器 Res-Attn，利用低秩注意力机制对查询、键和值进行投影，通过训练这个轻量级的注意力模块，实现了对新场景的自适应，并在一系列判别性和生成性任务中展示了与现有方法相比的优越性能。