Federated learning (FL) offers privacy-preserving decentralized machine
learning, optimizing models at edge clients without sharing private data.
Simultaneously, foundation models (FMs) have gained traction in the artificial
intelligence (AI) community due to their exceptional performance across various
tasks. However, integrating FMs into FL presents challenges, primarily due to
their substantial size and intensive resource requirements. This is especially
true when considering the resource heterogeneity in edge FL systems. We present
an adaptive framework for Resource-aware Federated Foundation Models (RaFFM) to
address these challenges. RaFFM introduces specialized model compression
algorithms tailored for FL scenarios, such as salient parameter prioritization
and high-performance subnetwork extraction. These algorithms enable dynamic
scaling of given transformer-based FMs to fit heterogeneous resource
constraints at the network edge during both FL's optimization and deployment
stages. Experimental results demonstrate that RaFFM shows significant
superiority in resource utilization efficiency and uses fewer resources to
deploy FMs to FL. Despite the lower resource consumption, target models
optimized by RaFFM achieve performance on par with traditional FL methods
applied to full-sized FMs. This is evident across tasks in both natural
language processing and computer vision domains.

资源感知的联邦基础模型（RaFFM）引入了针对联邦学习场景的特殊模型压缩算法，实现对给定的基于 Transformer 的基础模型的动态缩放，以适应网络边缘的异构资源约束。实验结果表明，RaFFM 在资源利用效率上显示出显著优势，并使用较少的资源将基础模型部署到联邦学习中，虽然资源消耗较低，但经 RaFFM 优化的目标模型在自然语言处理和计算机视觉领域的各项任务中表现出与应用于全尺寸基础模型的传统联邦学习方法相当的性能。