Foundation models have shown great success in natural language processing,
computer vision, and multimodal tasks. FMs have a large number of model
parameters, thus requiring a substantial amount of data to help optimize the
model during the training. Federated learning has revolutionized machine
learning by enabling collaborative learning from decentralized data while still
preserving the data privacy of clients. Despite the great benefits foundation
models can have empowered by federated learning, they face severe computation,
communication, and statistical challenges. In this paper, we propose a novel
two-stage federated learning algorithm called FedMS. A global expert is trained
in the first stage and a local expert is trained in the second stage to provide
better personalization. We construct a Mixture of Foundation Models (MoFM) with
these two experts and design a gate neural network with an inserted gate
adapter that joins the aggregation every communication round in the second
stage. To further adapt to edge computing scenarios with limited computational
resources, we design a novel Sparsely Activated LoRA (SAL) algorithm that
freezes the pre-trained foundation model parameters inserts low-rank adaptation
matrices into transformer blocks and activates them progressively during the
training. We employ extensive experiments to verify the effectiveness of FedMS,
results show that FedMS outperforms other SOTA baselines by up to 55.25% in
default settings.

提出了一种名为 FedMS 的新型两阶段联邦学习算法，使用全局专家和本地专家构建混合基准模型（MoFM），设计了一个门控神经网络插入门适配器来在第二阶段的每一轮通信中进行聚合，并使用一种新型 SAL 算法来适应具有有限计算资源的边缘计算场景，实验证明 FedMS 在默认设置下超过其他 SOTA 基准模型的性能达到 55.25% 的提升。