As large language models (LLMs) have become increasingly compute and memory
intensive, parameter-efficient fine-tuning (PEFT) methods are now a common
strategy to fine-tune LLMs. A popular PEFT method is Low-Rank Adapters (LoRA),
which adds trainable low-rank "adapters" to selected layers. Each adapter
consists of a low-rank matrix product, multiplicatively scaled by a
rank-dependent factor. This scaling factor, which divides adapters by a factor
of the rank, results in slowed learning and stunted performance for LoRA with
higher-rank adapters. Consequently, the use of LoRA in practice has generally
been limited to very low ranks. In this work, we study the impact of the
scaling factor on the learning process and prove that LoRA adapters should be
divided by a factor of the square root of the rank. Modifying LoRA with the
appropriate scaling factor, which we call the rank-stabilized LoRA (rsLoRA)
method, easily provides for a fine-tuning compute/performance trade-off, where
larger ranks can be used to trade off increased computational resources during
training for better fine-tuning performance, with no change in inference
computing cost.

通过修改 Low-Rank Adapters 方法中的缩放因子，我们提出了一种称为 rank-stabilized LoRA (rsLoRA) 的方法，可以在训练期间用更多的计算资源来换取更好的 fine-tuning 性能，并且在推理计算成本不变的情况下实现了 fine-tuning 计算性能的折中。