TL;DRLoRA-GA通过引入一种新的初始化方法,即梯度近似初始化(Low Rank Adaptation with Gradient Approximation),能够在保持效率和性能的同时达到与完全微调相当的收敛速度,进而显著提高模型性能和收敛速度。
Abstract
Fine-tuning large-scale pretrained models is prohibitively expensive in terms of computational and memory costs. LoRA, as one of the most popular parameter-efficient fine-tuning (PEFT) methods, offers a cost-effective alternative by fine-tuning an auxiliary low-rank model that has sign