Low Rank Adaptation (LoRA) has gained massive attention in the recent
generative AI research. One of the main advantages of LoRA is its ability to be
fused with pretrained models adding no overhead during inference. However, from
a mobile deployment standpoint, we can either avoid inference overhead in the
fused mode but lose the ability to switch adapters rapidly, or suffer
significant (up to 30% higher) inference latency while enabling rapid switching
in the unfused mode. LoRA also exhibits concept-loss when multiple adapters are
used concurrently. In this paper, we propose Sparse High Rank Adapters (SHiRA),
a new paradigm which incurs no inference overhead, enables rapid switching, and
significantly reduces concept-loss. Specifically, SHiRA can be trained by
directly tuning only 1-2% of the base model weights while leaving others
unchanged. This results in a highly sparse adapter which can be switched
directly in the fused mode. We further provide theoretical and empirical
insights on how high sparsity in SHiRA can aid multi-adapter fusion by reducing
concept loss. Our extensive experiments on LVMs and LLMs demonstrate that
finetuning only a small fraction of the parameters in the base model is
sufficient for many tasks while enabling both rapid switching and multi-adapter
fusion. Finally, we provide a latency- and memory-efficient SHiRA
implementation based on Parameter-Efficient Finetuning (PEFT) Library. This
implementation trains at nearly the same speed as LoRA while consuming lower
peak GPU memory, thus making SHiRA easy to adopt for practical use cases.

本文提出了基于稀疏高秩适配器 (SHiRA) 的新范式，通过直接调整基模型权重的 1-2% 来训练高度稀疏的适配器，以在融合模式下实现无推理开销、快速切换和显著降低概念损失的效果。对 LVMs 和 LLMs 的广泛实验表明，仅微调基模型的一小部分参数对许多任务已经足够，并且可以同时实现快速切换和多适配器融合。

稀疏高秩适配器

Sparse High Rank Adapters

Recent developments in Parameter-Efficient Fine-Tuning (PEFT) methods for
pretrained deep neural networks have captured widespread interest. In this
work, we study the enhancement of current PEFT methods by incorporating the
spectral information of pretrained weight matrices into the fine-tuning
procedure. We investigate two spectral adaptation mechanisms, namely additive
tuning and orthogonal rotation of the top singular vectors, both are done via
first carrying out Singular Value Decomposition (SVD) of pretrained weights and
then fine-tuning the top spectral space. We provide a theoretical analysis of
spectral fine-tuning and show that our approach improves the rank capacity of
low-rank adapters given a fixed trainable parameter budget. We show through
extensive experiments that the proposed fine-tuning model enables better
parameter efficiency and tuning performance as well as benefits multi-adapter
fusion. The code will be open-sourced for reproducibility.

研究了参数高效调优方法与预训练深度神经网络的关联，提出了将预训练权重矩阵的光谱信息融入调优过程的两种机制：加性调优和正交旋转顶级奇异向量。实验证明该方法能提高低秩适配器的秩能力，使得参数效率和调优性能得到提升，并对多适配器融合也有益处。