The interest in federated learning has surged in recent research due to its
unique ability to train a global model using privacy-secured information held
locally on each client. This paper pays particular attention to the issue of
client-side model heterogeneity, a pervasive challenge in the practical
implementation of FL that escalates its complexity. Assuming a scenario where
each client possesses varied memory storage, processing capabilities and
network bandwidth - a phenomenon referred to as system heterogeneity - there is
a pressing need to customize a unique model for each client. In response to
this, we present an effective and adaptable federated framework FedP3,
representing Federated Personalized and Privacy-friendly network Pruning,
tailored for model heterogeneity scenarios. Our proposed methodology can
incorporate and adapt well-established techniques to its specific instances. We
offer a theoretical interpretation of FedP3 and its locally
differential-private variant, DP-FedP3, and theoretically validate their
efficiencies.

本研究关注于联邦学习中的客户端模型异质性问题，并提出了适用于模型异质性场景的有效、适应性强的联邦框架 FedP3 及其差分隐私变体 DP-FedP3，并从理论上验证了它们的高效性。

FedP3：基于模型异构的联邦化个性化和隐私友好的网络剪枝

FedP3: Federated Personalized and Privacy-friendly Network Pruning under  Model Heterogeneity

Federated learning (FL) is a privacy-preserving machine learning paradigm in
which the server periodically aggregates local model parameters from clients
without assembling their private data. Constrained communication and
personalization requirements pose severe challenges to FL. Federated
distillation (FD) is proposed to simultaneously address the above two problems,
which exchanges knowledge between the server and clients, supporting
heterogeneous local models while significantly reducing communication overhead.
However, most existing FD methods require a proxy dataset, which is often
unavailable in reality. A few recent proxy-data-free FD approaches can
eliminate the need for additional public data, but suffer from remarkable
discrepancy among local knowledge due to client-side model heterogeneity,
leading to ambiguous representation on the server and inevitable accuracy
degradation. To tackle this issue, we propose a proxy-data-free FD algorithm
based on distributed knowledge congruence (FedDKC). FedDKC leverages
well-designed refinement strategies to narrow local knowledge differences into
an acceptable upper bound, so as to mitigate the negative effects of knowledge
incongruence. Specifically, from perspectives of peak probability and Shannon
entropy of local knowledge, we design kernel-based knowledge refinement (KKR)
and searching-based knowledge refinement (SKR) respectively, and theoretically
guarantee that the refined-local knowledge can satisfy an approximately-similar
distribution and be regarded as congruent. Extensive experiments conducted on
three common datasets demonstrate that our proposed FedDKC significantly
outperforms the state-of-the-art (Top-1 accuracy boosts by up to 4.38%, and
Top-5 accuracy boosts by up to 10.31%) on various heterogeneous settings while
evidently improving the convergence speed.

本文提出了一种基于分布式知识一致性的无代理数据 Federated Distillation 算法以减轻客户端模型异质性带来的知识不一致问题，并设计了优化策略以保证本地模型参数能够满足近似相似分布并被视为一致。实验表明该算法在多种异构设置下均明显优于现有方法并大幅提高了收敛速度。