Federated learning (FL) is a distributed machine learning paradigm in which the server periodically aggregates local model parameters from clients without assembling their private data. User-constrained communication bandwidth and the requirement for personalized models pose severe challenges to FL. Federated distillation (FD) is proposed to simultaneously address the two problems, which exchanges knowledge between the server and clients, supporting heterogeneous local models while significantly reducing communication overhead. However, most existing FD methods require a proxy dataset, which is often unavailable. Proxy-data-free FD approaches eliminate the need for additional public data beyond clients' private data, but suffer from remarkable discrepancy among local knowledge due to model heterogeneity, leading to ambiguous representation on the server and inevitable accuracy degradation. To tackle this issue, we propose a proxy-data-free FD algorithm based on distributed knowledge congruence (FedDKC). FedDKC leverages well-designed refinement strategies to narrow local knowledge differences into an acceptable upper bound to mitigate the negative effects of knowledge incongruence. Specifically, from perspectives of peak probability and Shannon entropy of local knowledge, we design kernel-based knowledge refinement (KKR) and searching-based knowledge refinement (SKR) respectively, and theoretically guarantee the refined-local knowledge can satisfy an approximately-similar distribution and be regarded as congruent. Extensive experiments conducted on three common datasets demonstrate that our proposed FedDKC method outperforms the state-of-the-art in 93.33% of comparisons, and achieves faster convergence without increasing communication overhead.

本文提出了一种基于分布式知识一致性的无代理数据 Federated Distillation 算法以减轻客户端模型异质性带来的知识不一致问题，并设计了优化策略以保证本地模型参数能够满足近似相似分布并被视为一致。实验表明该算法在多种异构设置下均明显优于现有方法并大幅提高了收敛速度。

探索无代理数据联邦蒸馏中分布式知识一致性