The performance of federated learning in neural networks is generally
influenced by the heterogeneity of the data distribution. For a well-performing
global model, taking a weighted average of the local models, as done by most
existing federated learning algorithms, may not guarantee consistency with
local models in the space of neural network maps. In this paper, we propose a
novel framework of federated learning equipped with the process of
decentralized knowledge distillation (FedDKD) (i.e., without data on the
server). The FedDKD introduces a module of decentralized knowledge distillation
(DKD) to distill the knowledge of the local models to train the global model by
approaching the neural network map average based on the metric of divergence
defined in the loss function, other than only averaging parameters as done in
literature. Numeric experiments on various heterogeneous datasets reveal that
FedDKD outperforms the state-of-the-art methods with more efficient
communication and training in a few DKD steps, especially on some extremely
heterogeneous datasets.

在神经网络中，联邦学习的性能通常受到数据分布的异构性的影响。本研究提出了一种新的联邦学习框架，配备了去中心化知识蒸馏（FedDKD）的过程，通过分布在本地的局部模型来训练全球模型，可以更有效地拟合神经网络映射平均值，来克服权重计算问题。实验证明，FedDKD 在一些极度异构的数据集上表现出更高的效率和更好的训练结果，优于现有的联邦学习算法。