With the increasing availability of Foundation Models, federated tuning has
garnered attention in the field of federated learning, utilizing data and
computation resources from multiple clients to collaboratively fine-tune
foundation models. However, in real-world federated scenarios, there often
exist a multitude of heterogeneous clients with varying computation and
communication resources, rendering them incapable of supporting the entire
model fine-tuning process. In response to this challenge, we propose a novel
federated tuning algorithm, FedRA. The implementation of FedRA is
straightforward and can be seamlessly integrated into any transformer-based
model without the need for further modification to the original model.
Specifically, in each communication round, FedRA randomly generates an
allocation matrix. For resource-constrained clients, it reorganizes a small
number of layers from the original model based on the allocation matrix and
fine-tunes using LoRA. Subsequently, the server aggregates the updated LoRA
parameters from the clients according to the current allocation matrix into the
corresponding layers of the original model. It is worth noting that FedRA also
supports scenarios where none of the clients can support the entire global
model, which is an impressive advantage. We conduct experiments on two
large-scale image datasets, DomainNet and NICO++, under various non-iid
settings. The results demonstrate that FedRA outperforms the compared methods
significantly. The source code is available at
https://github.com/leondada/FedRA.

使用 Foundation 模型，提出了一种新的联邦调优算法 FedRA，可在异构客户端中进行资源有限的模型微调，实验证明其性能明显优于其他方法。