Low-rank adaptation (LoRA) is a natural method for finetuning in
communication-constrained machine learning settings such as cross-device
federated learning. Prior work that has studied LoRA in the context of
federated learning has focused on improving LoRA's robustness to heterogeneity
and privacy. In this work, we instead consider techniques for further improving
communication-efficiency in federated LoRA. Unfortunately, we show that
centralized ML methods that improve the efficiency of LoRA through unstructured
pruning do not transfer well to federated settings. We instead study a simple
approach, \textbf{FLASC}, that applies sparsity to LoRA during communication
while allowing clients to locally fine-tune the entire LoRA module. Across four
common federated learning tasks, we demonstrate that this method matches the
performance of dense LoRA with up to $10\times$ less communication.
Additionally, despite being designed primarily to target communication, we find
that this approach has benefits in terms of heterogeneity and privacy relative
to existing approaches tailored to these specific concerns. Overall, our work
highlights the importance of considering system-specific constraints when
developing communication-efficient finetuning approaches, and serves as a
simple and competitive baseline for future work in federated finetuning.

在联邦学习中，通过应用稀疏性来提高 Low-rank adaptation (LoRA) 的通信效率，而不影响其性能，并且相对于现有方法具有更好的异质性和隐私保护。