Fine-tuning large-scale pre-trained models is prohibitively expensive in terms of computational and memory costs. Low-Rank Adaptation (LoRA), a popular Parameter-Efficient Fine-Tuning (PEFT) method, provides an efficient way to fine-tune models by optimizing only a low-rank matrix. Despite recent progress made in improving LoRA's performance, the connection between the LoRA optimization space and the original full parameter space is often overlooked. A solution that appears flat in the LoRA space may exist sharp directions in the full parameter space, potentially harming generalization performance. In this paper, we propose Flat-LoRA, an efficient approach that seeks a low-rank adaptation located in a flat region of the full parameter space.Instead of relying on the well-established sharpness-aware minimization approach, which can incur significant computational and memory burdens, we utilize random weight perturbation with a Bayesian expectation loss objective to maintain training efficiency and design a refined perturbation generation strategy for improved performance. Experiments on natural language processing and image classification tasks with various architectures demonstrate the effectiveness of our approach.

本研究解决了大规模预训练模型微调的高昂计算和内存成本问题，提出了Flat-LoRA方法，旨在寻找位于全参数空间平坦区域的低秩适应。通过随机权重扰动与贝叶斯期望损失目标相结合，Flat-LoRA在自然语言处理和图像分类任务中表现出了卓越的性能，提高了微调模型的效率和泛化能力。

平坦的LoRA：在平坦损失景观上的低秩适应