This paper introduces CURLoRA, a novel approach to fine-tuning large language models (LLMs) that leverages CUR matrix decomposition in the context of Low-Rank Adaptation (LoRA). Our method addresses two critical challenges in LLM fine-tuning: mitigating catastrophic forgetting during continual learning and reducing the number of trainable parameters. We propose a unique modification to the CUR decomposition process, utilizing inverted probabilities for column and row selection which acts as an implicit regularization, and initializing the $U$ matrix as a zero matrix, and only fine-tuning it. We demonstrate through experiments on multiple datasets that CURLoRA outperforms standard LoRA in mitigating catastrophic forgetting. It maintains model stability and performance across tasks while significantly reducing the number of trainable parameters. Our results show that CURLoRA achieves very good and stable task accuracy while maintaining base model's perplexity scores fixed compared to LoRA upon continual fine-tuning, particularly in scenarios with limited data.

本文提出了CURLoRA，一种利用CUR矩阵分解进行大规模语言模型微调的新方法，旨在解决灾难性遗忘与可训练参数减少两个问题。通过修改CUR分解过程，采用倒概率选择并初始化$U$矩阵为零矩阵，实验结果表明，CURLoRA在多个数据集上优于标准LoRA，能够在持续微调期间保持模型稳定性和性能，特别是在有限数据情况下。

CURLoRA：稳定LLM持续微调与灾难性遗忘的缓解