Low-rank adaptation (LoRA) is an attractive alternative of adapting full
weights for the federated fine-tuning of large pretrained models, which can
significantly reduce the memory and communication burden. In principle,
federated LoRA can provide an effective mean to allocate different resources to
each client by tuning ranks for each client, which can be useful in achieving a
better communication-performance tradeoff. We find, however, that the empirical
performance of LoRA is highly unstable with respect to such rank-heterogeneity,
severely limiting the applicability to the scenarios where it is desirable or
even required to allocate nonuniform communication bandwidth to each client due
to constrained total bandwidth. Our investigation reveals that the root cause
of this instability is the zero-padding-based aggregation strategy adopted in
conventional federated LoRA frameworks, which causes the information from high
rank clients to get diluted during the aggregation process. To address this
issue, we propose a new replication-based padding strategy, which allows us to
better leverage the information from clients with high-quality datasets. This
method ensures that valuable information from high rank clients is retained
during the aggregation process, accelerating the convergence speed and
enhancing the overall prediction quality of the global model.

通过提出一种新的基于复制的填充策略来解决传统的联邦低秩适应方法中的稳定性问题，以实现在联邦细调过程中更好地利用高质量数据集，加速收敛速度并提高全局模型的预测质量。

面向异构通信的联邦低秩自适应

Towards Federated Low-Rank Adaptation with Rank-Heterogeneous  Communication

In the current landscape of foundation model training, there is a significant
reliance on public domain data, which is nearing exhaustion according to recent
research. To further scale up, it is crucial to incorporate collaboration among
multiple specialized and high-quality private domain data sources. However, the
challenge of training models locally without sharing private data presents
numerous obstacles in data quality control. To tackle this issue, we propose a
data quality control pipeline for federated fine-tuning of foundation models.
This pipeline computes scores reflecting the quality of training data and
determines a global threshold for a unified standard, aiming for improved
global performance. Our experiments show that the proposed quality control
pipeline facilitates the effectiveness and reliability of the model training,
leading to better performance.

当前基础模型训练的情况表明，公共领域数据已接近枯竭，因此需要在多个专门化和高质量的私有领域数据源之间加强合作。为了解决本地训练模型而不共享私有数据所带来的质量控制问题，我们提出了一个针对基础模型的联邦微调的数据质量控制流程。该流程通过计算反映训练数据质量的分数，并确定一个统一标准的全局阈值，旨在提高整体性能。我们的实验证明，所提出的质量控制流程有助于模型训练的效果和可靠性，从而提高性能。