BriefGPT.xyz
Feb, 2024
NTK 条件下的 LoRA 训练无虚假局部极小值
LoRA Training in the NTK Regime has No Spurious Local Minima
HTML
PDF
Uijeong Jang, Jason D. Lee, Ernest K. Ryu
TL;DR
通过理论分析，在神经切线核（NTK）范式下，我们发现使用低秩适应（LoRA）进行微调有助于消除虚假的局部极小值点，从而使得梯度下降算法可以找到低秩解。
Abstract
low-rank adaptation
(LoRA) has become the standard approach for parameter-efficient
fine-tuning
of
large language models
(LLM), but our th
→