Low-rank adaptation (LoRA) is an attractive alternative of adapting full
weights for the federated fine-tuning of large pretrained models, which can
significantly reduce the memory and communication burden. In principle,
federated LoRA can provide an effective mean to allocate different resources to
each client by tuning ranks for each client, which can be useful in achieving a
better communication-performance tradeoff. We find, however, that the empirical
performance of LoRA is highly unstable with respect to such rank-heterogeneity,
severely limiting the applicability to the scenarios where it is desirable or
even required to allocate nonuniform communication bandwidth to each client due
to constrained total bandwidth. Our investigation reveals that the root cause
of this instability is the zero-padding-based aggregation strategy adopted in
conventional federated LoRA frameworks, which causes the information from high
rank clients to get diluted during the aggregation process. To address this
issue, we propose a new replication-based padding strategy, which allows us to
better leverage the information from clients with high-quality datasets. This
method ensures that valuable information from high rank clients is retained
during the aggregation process, accelerating the convergence speed and
enhancing the overall prediction quality of the global model.

通过提出一种新的基于复制的填充策略来解决传统的联邦低秩适应方法中的稳定性问题，以实现在联邦细调过程中更好地利用高质量数据集，加速收敛速度并提高全局模型的预测质量。