Fine-tuning Large Language Models (LLMs) has become a crucial technique for adapting pre-trained models to downstream tasks. However, the enormous size of LLMs poses significant challenges in terms of computational complexity and resource requirements. Low-Rank Adaptation (LoRA) has emerged as a promising solution. However, there exists a gap between the practical performance of low-rank adaptations and its theoretical optimum. In this work, we propose eXtreme Gradient Boosting LoRA (XGBLoRA), a novel framework that bridges this gap by leveraging the power of ensemble learning. Inspired by gradient boosting, XGBLoRA iteratively learns and merges a sequence of LoRA adaptations to refine model predictions. It achieves better performance than the standard LoRA, while enjoying the computational efficiency of rank-1 adaptations. We provide theoretical analysis to show the convergence and optimality of our approach, and conduct extensive experiments on a range of natural language processing tasks. The results demonstrate that XGBLoRA consistently outperforms standard LoRA and achieves performance comparable to full fine-tuning with significantly fewer trainable parameters. This work advances parameter-efficient fine-tuning for LLMs, and offers a promising solution for adapting LLMs to downstream tasks while optimizing performance and efficiency.

本研究解决了微调大型语言模型（LLMs）时存在的计算复杂性和资源需求问题，尤其是低秩适应与理论最佳性能之间的差距。提出的极端梯度提升LoRA（XGBLoRA）框架利用集成学习的优势，通过迭代学习和合并LoRA适应来优化模型预测，最终实现了在计算效率上超越标准LoRA，同时性能与全面微调相媲美。此研究推动了大型语言模型的高效微调，为模型在下游任务中的适应提供了更优的解决方案。

少即是多：极端梯度提升Rank-1自适应用于高效微调大型语言模型