Federated Learning (FL) has recently been applied to the parameter-efficient
fine-tuning of Large Language Models (LLMs). While promising, it raises
significant challenges due to the heterogeneous resources and data
distributions of clients.This study introduces FlexLoRA, a simple yet effective
aggregation scheme for LLM fine-tuning, which mitigates the "buckets effect" in
traditional FL that restricts the potential of clients with ample resources by
tying them to the capabilities of the least-resourced participants. FlexLoRA
allows for dynamic adjustment of local LoRA ranks, fostering the development of
a global model imbued with broader, less task-specific knowledge. By
synthesizing a full-size LoRA weight from individual client contributions and
employing Singular Value Decomposition (SVD) for weight redistribution,
FlexLoRA fully leverages heterogeneous client resources. Involving over 1,600
clients performing diverse NLP tasks, our experiments validate the efficacy of
FlexLoRA, with the federated global model achieving up to a 3.1% average
improvement in downstream NLP task performance. FlexLoRA's practicality is
further underscored by its seamless integration with existing LoRA-based FL
methods and theoretical analysis, offering a path toward scalable,
privacy-preserving federated tuning for LLMs.

FlexLoRA 是一种用于 LLM fine-tuning 的聚合方案，通过动态调整本地 LoRA 等级并使用奇异值分解（SVD）进行权重重新分配，充分利用了异构客户端资源。在涉及超过 1,600 个客户端执行多样化的自然语言处理（NLP）任务的实验中，证实了 FlexLoRA 的有效性，联邦全局模型在下游 NLP 任务性能方面平均提高了 3.1%。FlexLoRA 的实用性进一步强调了其与现有的基于 LoRA 的联邦学习方法的无缝集成和理论分析，提供了一种可扩展的、隐私保护的 LLM 联邦调整路径。

异构语言任务和客户资源下的分布式大型语言模型微调

Federated Fine-tuning of Large Language Models under Heterogeneous  Language Tasks and Client Resources

Common crowdsourcing systems average estimates of a latent quantity of
interest provided by many crowdworkers to produce a group estimate. We develop
a new approach -- just-predict-others -- that leverages self-supervised
learning and a novel aggregation scheme. This approach adapts weights assigned
to crowdworkers based on estimates they provided for previous quantities. When
skills vary across crowdworkers or their estimates correlate, the weighted sum
offers a more accurate group estimate than the average. Existing algorithms
such as expectation maximization can, at least in principle, produce similarly
accurate group estimates. However, their computational requirements become
onerous when complex models, such as neural networks, are required to express
relationships among crowdworkers. Just-predict-others accommodates such
complexity as well as many other practical challenges. We analyze the efficacy
of just-predict-others through theoretical and computational studies. Among
other things, we establish asymptotic optimality as the number of engagements
per crowdworker grows.

利用自监督学习和新颖的聚合机制，just-predict-others 方法在面对具有不同技能水平和相关估计的众包工作者时，比传统的众包系统更能提供准确的群体估计；并通过理论和计算研究验证 just-predict-others 的效力，证明随着每个众包工作者参与次数的增加，该方法在渐近优化方面表现优秀。