In this paper, we introduce a subspace-inspired Low-Rank Adaptation (LoRA)
method, which is computationally efficient, easy to implement, and readily
applicable to large language, multimodal, and diffusion models. Initially, we
equivalently decompose the weights of LoRA into two subspaces, and find that
simply mixing them can enhance performance. To study such a phenomenon, we
revisit it through a fine-grained subspace lens, showing that such modification
is equivalent to employing a fixed mixer to fuse the subspaces. To be more
flexible, we jointly learn the mixer with the original LoRA weights, and term
the method Mixture-of-Subspaces LoRA (MoSLoRA). MoSLoRA consistently
outperforms LoRA on tasks in different modalities, including commonsense
reasoning, visual instruction tuning, and subject-driven text-to-image
generation, demonstrating its effectiveness and robustness. Codes are available
at \href{this https URL}{github}.

本文介绍了一种基于子空间的低秩适应方法（LoRA），该方法在计算效率、实施简易性和适用性方面具有优势，可广泛应用于大型语言、多模态和扩散模型；通过子空间的微观视角，我们发现混合两个子空间可以提升性能，并通过与原始 LoRA 权重共同学习混合器，进一步灵活性地实现该方法，称之为混合子空间 LoRA（MoSLoRA）；在不同模态的任务中，包括常识推理、视觉指令微调和基于主题的文本到图像生成，MoSLoRA 始终表现出色，验证了其有效性和稳健性。