Transformers have achieved remarkable success in various machine-learning
tasks, prompting their widespread adoption. In this paper, we explore their
application in the context of federated learning (FL), with a particular focus
on heterogeneous scenarios where individual clients possess diverse local
datasets. To meet the computational and communication demands of FL, we
leverage pre-trained Transformers and use an efficient prompt-tuning strategy.
Our strategy introduces the concept of learning both shared and group prompts,
enabling the acquisition of universal knowledge and group-specific knowledge
simultaneously. Additionally, a prompt selection module assigns personalized
group prompts to each input, aligning the global model with the data
distribution of each client. This approach allows us to train a single global
model that can automatically adapt to various local client data distributions
without requiring local fine-tuning. In this way, our proposed method
effectively bridges the gap between global and personalized local models in
Federated Learning and surpasses alternative approaches that lack the
capability to adapt to previously unseen clients. The effectiveness of our
approach is rigorously validated through extensive experimentation and ablation
studies.

在联邦学习中，我们利用预训练的 Transformer 和高效的提示调整策略，通过引入学习共享和组特定提示的概念，使全局模型能够自动适应各种局部客户数据分布，从而有效地弥合全局和个性化本地模型之间的差距，并超越以往无法适应之前未见客户的其他方法。