Knowledge distillation (KD) can enable collaborative learning among
distributed clients that have different model architectures and do not share
their local data and model parameters with others. Each client updates its
local model using the average model output/feature of all client models as the
target, known as federated KD. However, existing federated KD methods often do
not perform well when clients' local models are trained with heterogeneous
local datasets. In this paper, we propose Federated knowledge distillation
enabled by Adversarial Learning (FedAL) to address the data heterogeneity among
clients. First, to alleviate the local model output divergence across clients
caused by data heterogeneity, the server acts as a discriminator to guide
clients' local model training to achieve consensus model outputs among clients
through a min-max game between clients and the discriminator. Moreover,
catastrophic forgetting may happen during the clients' local training and
global knowledge transfer due to clients' heterogeneous local data. Towards
this challenge, we design the less-forgetting regularization for both local
training and global knowledge transfer to guarantee clients' ability to
transfer/learn knowledge to/from others. Experimental results show that FedAL
and its variants achieve higher accuracy than other federated KD baselines.

本文提出了一种名为 FedAL 的基于对抗学习的联邦知识蒸馏方法，以解决不同客户端的数据异构性问题，通过使用服务器作为鉴别器指导客户端的本地模型训练，实现客户端之间的共识模型输出，并通过较少遗忘的正则化方法保证客户端之间的知识传递和学习能力。实验证明，FedAL 及其变体比其他联邦知识蒸馏方法具有更高的准确率。