Federated Learning (FL) is often impeded by communication overhead issues. Prompt tuning, as a potential solution, has been introduced to only adjust a few trainable parameters rather than the whole model. However, current single-modality prompt tuning approaches fail to comprehensively portray local clients' data. To overcome this limitation, we present Twin Prompt Federated learning (TPFL), a pioneering solution that integrates both visual and textual modalities, ensuring a more holistic representation of local clients' data characteristics. Furthermore, in order to tackle the data heterogeneity issues, we introduce the Augmented TPFL (ATPFL) employing the contrastive learning to TPFL, which not only enhances the global knowledge acquisition of client models but also fosters the development of robust, compact models. The effectiveness of TPFL and ATPFL is substantiated by our extensive evaluations, consistently showing superior performance compared to all baselines.

联邦学习经常受到通信开销问题的阻碍。为了解决这个问题，我们提出了双提示联邦学习（TPFL），它整合了视觉和文本模态以更全面地描绘本地客户端数据特征。此外，为了应对数据异质性问题，我们引入了增强型TPFL（ATPFL），并采用对比学习技术，不仅增强了客户端模型的全局知识获取能力，还促进了稳健、紧凑模型的发展。我们的详尽评估结果表明，TPFL和ATPFL相比于所有基准模型始终表现出更优异的性能。

联邦学习中的包容性数据表示：将文本与视觉提示集成的新方法