The recent trend in scaling language models has led to a growing demand for
parameter-efficient tuning (PEFT) methods such as LoRA (Low-Rank Adaptation).
LoRA consistently matches or surpasses the full fine-tuning baseline with fewer
parameters. However, handling numerous task-specific or user-specific LoRA
modules on top of a base model still presents significant storage challenges.
To address this, we introduce LoRA-XS (Low-Rank Adaptation with eXtremely Small
number of parameters), a novel approach leveraging Singular Value Decomposition
(SVD) for parameter-efficient fine-tuning. LoRA-XS introduces a small r x r
weight matrix between frozen LoRA matrices, which are constructed by SVD of the
original weight matrix. Training only r x r weight matrices ensures
independence from model dimensions, enabling more parameter-efficient
fine-tuning, especially for larger models. LoRA-XS achieves a remarkable
reduction of trainable parameters by over 100x in 7B models compared to LoRA.
Our benchmarking across various scales, including GLUE, GSM8k, and MATH
benchmarks, shows that our approach outperforms LoRA and recent
state-of-the-art approaches like VeRA in terms of parameter efficiency while
maintaining competitive performance.

介绍了一种新的参数高效微调方法 LoRA-XS，利用奇异值分解（SVD）在参数高效微调方面取得显著的结果，特别是在较大模型上，比 LoRA 和 VeRA 等最新方法更具参数效率同时保持竞争性能。