BriefGPT.xyz
Feb, 2024
利用黎曼预条件的LoRA方法对基础模型进行微调
Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models
HTML
PDF
Fangzhao Zhang, Mert Pilanci
TL;DR
通过引入Riemannian预处理器,研究Low Rank Adaptation(LoRA)微调过程的增强,实验结果表明,使用我们的预处理器可以显著提高SGD和AdamW的收敛性和稳定性。
Abstract
In this work we study the enhancement of
low rank adaptation
(LoRA) fine-tuning procedure by introducing a Riemannian
preconditioner
in its
optim
→