As a privacy-preserving collaborative machine learning paradigm, federated
learning (FL) has attracted significant interest from academia and the industry
alike. To allow each data owner (a.k.a., FL clients) to train a heterogeneous
and personalized local model based on its local data distribution, system
resources and requirements on model structure, the field of model-heterogeneous
personalized federated learning (MHPFL) has emerged. Existing MHPFL approaches
either rely on the availability of a public dataset with special
characteristics to facilitate knowledge transfer, incur high computation and
communication costs, or face potential model leakage risks. To address these
limitations, we propose a model-heterogeneous personalized Federated learning
approach based on feature Extractor Sharing (pFedES). It incorporates a small
homogeneous feature extractor into each client's heterogeneous local model.
Clients train them via the proposed iterative learning method to enable the
exchange of global generalized knowledge and local personalized knowledge. The
small local homogeneous extractors produced after local training are uploaded
to the FL server and for aggregation to facilitate easy knowledge sharing among
clients. We theoretically prove that pFedES can converge over wall-to-wall
time. Extensive experiments on two real-world datasets against six
state-of-the-art methods demonstrate that pFedES builds the most accurate
model, while incurring low communication and computation costs. Compared with
the best-performing baseline, it achieves 1.61% higher test accuracy, while
reducing communication and computation costs by 99.6% and 82.9%, respectively.

提出了基于特征提取器共享的异构个性化联邦学习方法 (pFedES)，该方法在不同客户端的本地模型中引入小型同质特征提取器，通过迭代学习方法进行训练，实现全局泛化知识和本地个性化知识的交换和共享，其理论上证明了 pFedES 能够在墙时收敛。在两个真实数据集上的广泛实验中，与六种最先进的方法相比，pFedES 构建了最准确的模型，同时通信和计算成本较低，与最佳基准相比，测试准确性提高了 1.61%，通信和计算成本分别降低了 99.6% 和 82.9%。

pFedES: 模型异构个性化的联邦学习与特征提取器共享

pFedES: Model Heterogeneous Personalized Federated Learning with Feature  Extractor Sharing

Federated learning (FL) enables multiple participants to collaboratively
train machine learning models using decentralized data sources, alleviating
privacy concerns that arise from directly sharing local data. However, the lack
of model privacy protection in FL becomes an unneglectable challenge,
especially when people want to federally finetune models based on a proprietary
large language model. In this study, we propose a novel FL training approach
that accomplishes information exchange among participants via tunable soft
prompts. These soft prompts, updated and transmitted between the server and
clients, assume the role of the global model parameters and serve as messengers
to deliver useful knowledge from the local data and global model. As the global
model itself is not required to be shared and the local training is conducted
based on an auxiliary model with fewer parameters than the global model, the
proposed approach provides protection for the global model while reducing
communication and computation costs in FL. Extensive experiments show the
effectiveness of the proposed approach compared to several baselines. We have
released the source code at
https://github.com/alibaba/FederatedScope/tree/fedsp/federatedscope/nlp/fedsp.

通过可调整的软提示实现参与者之间的信息交流，以在损失较少的全局模型的基础上保护全局模型，减少联邦学习中的通信和计算成本。