Federated learning enables multiple clients to collaboratively learn a global
model by periodically aggregating the clients' models without transferring the
local data. However, due to the heterogeneity of the system and data, many
approaches suffer from the "client-drift" issue that could significantly slow
down the convergence of the global model training. As clients perform local
updates on heterogeneous data through heterogeneous systems, their local models
drift apart. To tackle this issue, one intuitive idea is to guide the local
model training by the global teachers, i.e., past global models, where each
client learns the global knowledge from past global models via adaptive
knowledge distillation techniques. Coming from these insights, we propose a
novel approach for heterogeneous federated learning, namely FedGKD, which fuses
the knowledge from historical global models for local training to alleviate the
"client-drift" issue. In this paper, we evaluate FedGKD with extensive
experiments on various CV/NLP datasets (i.e., CIFAR-10/100, Tiny-ImageNet, AG
News, SST5) and different heterogeneous settings. The proposed method is
guaranteed to converge under common assumptions, and achieves superior
empirical accuracy in fewer communication runs than five state-of-the-art
methods.

该论文提出一种名为 FedGKD 的新方法，通过融合历史全局模型的知识进行本地训练，解决异构联邦学习中的客户端漂移问题，并在各种计算机视觉和自然语言处理数据集上进行广泛的实验和评估，达到了优于其他五种方法的结果。