This study introduces an approach to optimize Parameter Efficient Fine Tuning (PEFT) for Pretrained Language Models (PLMs) by implementing a Shared Low Rank Adaptation (ShareLoRA). By strategically deploying ShareLoRA across different layers and adapting it for the Query, Key, and Value components of self-attention layers, we achieve a substantial reduction in the number of training parameters and memory usage. Importantly, ShareLoRA not only maintains model performance but also exhibits robustness in both classification and generation tasks across a variety of models, including RoBERTa, GPT-2, LLaMA and LLaMA2. It demonstrates superior transfer learning capabilities compared to standard LoRA applications and mitigates overfitting by sharing weights across layers. Our findings affirm that ShareLoRA effectively boosts parameter efficiency while ensuring scalable and high-quality performance across different language model architectures.

通过实施共享低秩适应（ShareLoRA）的方式，本研究介绍了一种优化预训练语言模型（PLMs）的参数有效微调（PEFT）的方法。在不同层级上策略性地部署ShareLoRA，并对self-attention层的Query、Key和Value组件进行适应性调整，我们实现了训练参数数量和内存使用的大幅减少。同时，ShareLoRA不仅在RoBERTa、GPT-2、LLaMA和LLaMA2等多种模型上保持了模型性能，还在分类和生成任务中表现出鲁棒性。相比标准的LoRA应用，它表现出卓越的迁移学习能力，并通过在层级间共享权重来减轻过拟合。我们的发现证明，ShareLoRA能够有效提升参数效率，同时在不同的语言模型架构上保证可扩展和高质量的性能。

ShareLoRA: 共享低秩调整的参数高效和鲁棒大型语言模型微调