The rapid advancements in large language models (LLMs) have revolutionized natural language processing, creating an increased need for efficient, task-specific fine-tuning methods. Traditional fine-tuning of LLMs involves updating a large number of parameters, which is computationally expensive and memory-intensive. Low-Rank Adaptation (LoRA) has emerged as a promising solution, enabling parameter-efficient fine-tuning by reducing the number of trainable parameters. However, while LoRA reduces the number of trainable parameters, LoRA modules still create significant storage challenges. We propose LoRA-Mini, an optimized adaptation of LoRA that improves parameter efficiency by splitting low-rank matrices into four parts, with only the two inner matrices being trainable. This approach achieves upto a 20x reduction compared to standard LoRA in the number of trainable parameters while preserving performance levels comparable to standard LoRA, addressing both computational and storage efficiency in LLM fine-tuning.

本研究针对大语言模型（LLMs）在特定任务微调中面临的高计算和存储成本问题，提出了一种新的LoRA-Mini方法。该方法通过将低秩矩阵分为四个部分，仅训练其中两个内矩阵，从而实现参数效率提升，训练参数数量最多减少至标准LoRA的20倍，同时保持了与标准LoRA相当的性能，显著改善了LLMs微调的计算和存储效率。

LoRA-Mini：适应矩阵分解与选择性训练