Federated Learning (FL) has recently been applied to the parameter-efficient
fine-tuning of Large Language Models (LLMs). While promising, it raises
significant challenges due to the heterogeneous resources and data
distributions of clients.This study introduces FlexLoRA, a simple yet effective
aggregation scheme for LLM fine-tuning, which mitigates the "buckets effect" in
traditional FL that restricts the potential of clients with ample resources by
tying them to the capabilities of the least-resourced participants. FlexLoRA
allows for dynamic adjustment of local LoRA ranks, fostering the development of
a global model imbued with broader, less task-specific knowledge. By
synthesizing a full-size LoRA weight from individual client contributions and
employing Singular Value Decomposition (SVD) for weight redistribution,
FlexLoRA fully leverages heterogeneous client resources. Involving over 1,600
clients performing diverse NLP tasks, our experiments validate the efficacy of
FlexLoRA, with the federated global model achieving up to a 3.1% average
improvement in downstream NLP task performance. FlexLoRA's practicality is
further underscored by its seamless integration with existing LoRA-based FL
methods and theoretical analysis, offering a path toward scalable,
privacy-preserving federated tuning for LLMs.

FlexLoRA 是一种用于 LLM fine-tuning 的聚合方案，通过动态调整本地 LoRA 等级并使用奇异值分解（SVD）进行权重重新分配，充分利用了异构客户端资源。在涉及超过 1,600 个客户端执行多样化的自然语言处理（NLP）任务的实验中，证实了 FlexLoRA 的有效性，联邦全局模型在下游 NLP 任务性能方面平均提高了 3.1%。FlexLoRA 的实用性进一步强调了其与现有的基于 LoRA 的联邦学习方法的无缝集成和理论分析，提供了一种可扩展的、隐私保护的 LLM 联邦调整路径。