Federated Learning (FL) enables multiple machines to collaboratively train a
machine learning model without sharing of private training data. Yet,
especially for heterogeneous models, a key bottleneck remains the transfer of
knowledge gained from each client model with the server. One popular method,
FedDF, uses distillation to tackle this task with the use of a common, shared
dataset on which predictions are exchanged. However, in many contexts such a
dataset might be difficult to acquire due to privacy and the clients might not
allow for storage of a large shared dataset. To this end, in this paper, we
introduce a new method that improves this knowledge distillation method to only
rely on a single shared image between clients and server. In particular, we
propose a novel adaptive dataset pruning algorithm that selects the most
informative crops generated from only a single image. With this, we show that
federated learning with distillation under a limited shared dataset budget
works better by using a single image compared to multiple individual ones.
Finally, we extend our approach to allow for training heterogeneous client
architectures by incorporating a non-uniform distillation schedule and
client-model mirroring on the server side.

通过改进知识蒸馏方法和引入一种自适应数据集修剪算法，我们提出一种新方法，在有限的共享数据集预算下，使用一幅图像成功地进行联邦学习，同时还可以训练异构的客户端架构。