Model-heterogeneous personalized federated learning (MHPFL) enables FL
clients to train structurally different personalized models on non-independent
and identically distributed (non-IID) local data. Existing MHPFL methods focus
on achieving client-level personalization, but cannot address batch-level data
heterogeneity. To bridge this important gap, we propose a model-heterogeneous
personalized Federated learning approach with Adaptive Feature Mixture
(pFedAFM) for supervised learning tasks. It consists of three novel designs: 1)
A sharing global homogeneous small feature extractor is assigned alongside each
client's local heterogeneous model (consisting of a heterogeneous feature
extractor and a prediction header) to facilitate cross-client knowledge fusion.
The two feature extractors share the local heterogeneous model's prediction
header containing rich personalized prediction knowledge to retain personalized
prediction capabilities. 2) An iterative training strategy is designed to
alternately train the global homogeneous small feature extractor and the local
heterogeneous large model for effective global-local knowledge exchange. 3) A
trainable weight vector is designed to dynamically mix the features extracted
by both feature extractors to adapt to batch-level data heterogeneity.
Theoretical analysis proves that pFedAFM can converge over time. Extensive
experiments on 2 benchmark datasets demonstrate that it significantly
outperforms 7 state-of-the-art MHPFL methods, achieving up to 7.93% accuracy
improvement while incurring low communication and computation costs.

基于模型异构的个性化联邦学习通过自适应特征混合（pFedAFM）实现了训练不同结构个性化模型的联邦学习客户端，以应对非独立同分布的本地数据。

pFedAFM：异构联邦学习中基于批次的个性化自适应特征混合

pFedAFM: Adaptive Feature Mixture for Batch-Level Personalization in  Heterogeneous Federated Learning

Federated learning (FL) is widely employed for collaborative training on
decentralized data but faces challenges like data, system, and model
heterogeneity. This prompted the emergency of model-heterogeneous personalized
federated learning (MHPFL). However, concerns persist regarding data and model
privacy, model performance, communication, and computational costs in current
MHPFL methods. To tackle these concerns, we propose a novel model-heterogeneous
personalized Federated learning algorithm (FedMoE) with the Mixture of Experts
(MoE), renowned for enhancing large language models (LLMs). It assigns a shared
homogeneous small feature extractor and a local gating network for each
client's local heterogeneous large model. (1) During local training, the local
heterogeneous model's feature extractor acts as a local expert for personalized
feature (representation) extraction, while the shared homogeneous small feature
extractor serves as a global expert for generalized feature extraction. The
local gating network produces personalized weights for extracted
representations from both experts on each data sample. The three models form a
local heterogeneous MoE. The weighted mixed representation fuses global
generalized and local personalized features and is processed by the local
heterogeneous large model's header with personalized prediction information for
output. The MoE and prediction header are updated synchronously. (2) The
trained local homogeneous small feature extractors are sent to the server for
cross-client information fusion via aggregation. Briefly, FedMoE first enhances
local model personalization at a fine-grained data level while supporting model
heterogeneity.

通过引入混合专家方法，我们提出了一种新颖的模型异构个性化联邦学习算法（FedMoE），用于解决当前模型异构个性化联邦学习方法中的数据和模型隐私、模型性能、通信和计算成本等问题。该算法通过在本地训练过程中结合全局和本地特征来提高模型的个性化能力，并通过信息聚合实现跨客户端的模型参数融合。

FedMoE：模型异构个性化联邦学习中基于数据的个性化

FedMoE: Data-Level Personalization with Mixture of Experts for  Model-Heterogeneous Personalized Federated Learning

As a privacy-preserving collaborative machine learning paradigm, federated
learning (FL) has attracted significant interest from academia and the industry
alike. To allow each data owner (a.k.a., FL clients) to train a heterogeneous
and personalized local model based on its local data distribution, system
resources and requirements on model structure, the field of model-heterogeneous
personalized federated learning (MHPFL) has emerged. Existing MHPFL approaches
either rely on the availability of a public dataset with special
characteristics to facilitate knowledge transfer, incur high computation and
communication costs, or face potential model leakage risks. To address these
limitations, we propose a model-heterogeneous personalized Federated learning
approach based on feature Extractor Sharing (pFedES). It incorporates a small
homogeneous feature extractor into each client's heterogeneous local model.
Clients train them via the proposed iterative learning method to enable the
exchange of global generalized knowledge and local personalized knowledge. The
small local homogeneous extractors produced after local training are uploaded
to the FL server and for aggregation to facilitate easy knowledge sharing among
clients. We theoretically prove that pFedES can converge over wall-to-wall
time. Extensive experiments on two real-world datasets against six
state-of-the-art methods demonstrate that pFedES builds the most accurate
model, while incurring low communication and computation costs. Compared with
the best-performing baseline, it achieves 1.61% higher test accuracy, while
reducing communication and computation costs by 99.6% and 82.9%, respectively.

提出了基于特征提取器共享的异构个性化联邦学习方法 (pFedES)，该方法在不同客户端的本地模型中引入小型同质特征提取器，通过迭代学习方法进行训练，实现全局泛化知识和本地个性化知识的交换和共享，其理论上证明了 pFedES 能够在墙时收敛。在两个真实数据集上的广泛实验中，与六种最先进的方法相比，pFedES 构建了最准确的模型，同时通信和计算成本较低，与最佳基准相比，测试准确性提高了 1.61%，通信和计算成本分别降低了 99.6% 和 82.9%。