We investigate parameter-efficient fine-tuning (PEFT) methods that can provide good accuracy under limited computational and memory budgets in the context of large language models (LLMs). We present a new PEFT method called Robust Adaptation (RoSA) inspired by robust principal component analysis (PCA) that jointly trains $\textit{low-rank}$ and $\textit{highly-sparse}$ components on top of a set of fixed pretrained weights to efficiently approximate the performance of a full-fine-tuning (FFT) solution. Across a series of challenging generative tasks such as grade-school math and SQL query generation, which require fine-tuning for good performance, we show that RoSA outperforms both LoRA and pure sparse fine-tuning, at the same parameter budget. We provide system support for RoSA to complement the training algorithm, specifically in the form of sparse GPU kernels which enable memory- and computationally-efficient training. Our code will be made available at https://github.com/IST-DASLab/RoSA}{\texttt{https://github.com/IST-DASLab/RoSA

我们研究了能够在计算和内存有限的情况下提供良好准确度的参数高效调整方法（PEFT），我们提出了一种新的PEFT方法称为Robust Adaptation（RoSA），通过在一组固定的预训练权重之上联合训练低秩和高度稀疏的组件，有效地逼近全精调（FFT）解决方案的性能，在需要进行精细调整以获得良好性能的挑战性生成任务中，如小学数学和SQL查询生成，我们展示了RoSA优于LoRA和纯稀疏调整在相同参数预算下的性能。我们为RoSA提供系统支持，以在训练算法中补充，具体为稀疏GPU内核，实现内存和计算上的高效训练。我们的代码将在https://github.com/IST-DASLab/RoSA上提供。

RoSA：鲁棒适应实现准确的参数高效微调