Federated Prompt Learning (FPL) incorporates large pre-trained
Vision-Language models (VLM) into federated learning through prompt tuning. The
transferable representations and remarkable generalization capacity of VLM make
them highly compatible with the integration of federated learning. Addressing
data heterogeneity in federated learning requires personalization, but
excessive focus on it across clients could compromise the model's ability to
generalize effectively. To preserve the impressive generalization capability of
VLM, it is crucial to strike a balance between personalization and
generalization in FPL. To tackle this challenge, we proposed Federated Prompt
Learning with CLIP Generalization and low-rank Personalization (FedPGP), which
employs pre-trained CLIP to provide knowledge-guidance on the global prompt for
improved generalization and incorporates a low-rank adaptation term to
personalize the global prompt. Further, FedPGP integrates a prompt-wise
contrastive loss to achieve knowledge guidance and personalized adaptation
simultaneously, enabling a harmonious balance between personalization and
generalization in FPL. We conduct extensive experiments on various datasets to
explore base-to-novel generalization in both category-level and domain-level
scenarios with heterogeneous data, showing the superiority of FedPGP in
balancing generalization and personalization.

通过使用预先训练的视觉语言模型（VLM）调整提示，联邦提示学习（FPL）将大规模预训练的视觉语言模型（VLM）整合到联邦学习中，以达到在个性化和泛化之间取得平衡的目标。