LoRA achieves remarkable resource efficiency and comparable performance when
adapting LLMs for specific tasks. Since ChatGPT demonstrated superior
performance on various tasks, there has been a growing desire to adapt one
model for all tasks. However, the explicit low-rank of LoRA limits the
adaptation performance in complex multi-task scenarios. LoRA is dominated by a
small number of top singular vectors while fine-tuning decomposes into a set of
less important unitary transforms. In this paper, we propose MultiLoRA for
better multi-task adaptation by reducing the dominance of top singular vectors
observed in LoRA. MultiLoRA scales LoRA modules horizontally and change
parameter initialization of adaptation matrices to reduce parameter dependency,
thus yields more balanced unitary subspaces. We unprecedentedly construct
specialized training data by mixing datasets of instruction follow, natural
language understanding, world knowledge, to cover semantically and
syntactically different samples. With only 2.5% of additional parameters,
MultiLoRA outperforms single LoRA counterparts and fine-tuning on multiple
benchmarks and model scales. Further investigation into weight update matrices
of MultiLoRA exhibits reduced dependency on top singular vectors and more
democratic unitary transform contributions.

提出了 MultiLoRA，通过减少 LoRA 中观察到的顶层奇异向量的主导性，通过水平扩展 LoRA 模块和改变适应矩阵的参数初始化来减少参数依赖性，从而产生更平衡的单元子空间，从而实现更好的多任务适应性。MultiLoRA 在多个基准和模型规模上优于单个 LoRA 对应项和微调，仅需额外 2.5％的参数。对 MultiLoRA 的权重更新矩阵进行进一步研究表明，其对顶层奇异向量的依赖性减小，单元变换贡献更加均衡。