BriefGPT.xyz
Feb, 2024
LoRA+：大型模型的高效低秩自适应
LoRA+: Efficient Low Rank Adaptation of Large Models
HTML
PDF
Soufiane Hayou, Nikhil Ghosh, Bin Yu
TL;DR
通过设置适当的比率为LoRA适配器矩阵A和B设置不同的学习率，我们提出了一种名为LoRA$+$的算法，解决了LoRA的次优问题同时提高了性能（1-2％改进）和微调速度（最高约2倍速度提升）的问题。
Abstract
In this paper, we show that
low rank adaptation
(
lora
) as originally introduced in Hu et al. (2021) leads to suboptimal
finetuning
of mode
→