Low-rank adaptation (LoRA) is one of the most popular task-specific
parameter-efficient fine-tuning (PEFT) methods on pre-trained language models
for its good performance and computational efficiency. LoRA injects a product
of two trainable rank decomposition matrices over the top of each frozen
pre-trained model module. However, when applied in the setting of
privacy-preserving federated learning (FL), LoRA may become unstable due to the
following facts: 1) the effects of data heterogeneity and multi-step local
updates are non-negligible, 2) additive noise enforced on updating gradients to
guarantee differential privacy (DP) can be amplified and 3) the final
performance is susceptible to hyper-parameters. A key factor leading to these
phenomena is the discordance between jointly optimizing the two low-rank
matrices by local clients and separately aggregating them by the central
server. Thus, this paper proposes an efficient and effective version of LoRA,
Federated Freeze A LoRA (FFA-LoRA), to alleviate these challenges and further
halve the communication cost of federated fine-tuning LLMs. The core idea of
FFA-LoRA is to fix the randomly initialized non-zero matrices and only
fine-tune the zero-initialized matrices. Compared to LoRA, FFA-LoRA is
motivated by practical and theoretical benefits in privacy-preserved FL. Our
experiments demonstrate that FFA-LoRA provides more consistent performance with
better computational efficiency over vanilla LoRA in various FL tasks.

在隐私保护联邦学习中，本文提出了一种高效且有效的低秩适应方法 FFA-LoRA，通过固定非零矩阵并仅微调零矩阵，缓解了数据异构性、差分隐私增强噪声放大以及超参数敏感性等挑战，同时将通信成本减半，并在各种联邦学习任务中展现了更一致的性能和更好的计算效率。