Despite Federated Learning (FL)'s trend for learning machine learning models
in a distributed manner, it is susceptible to performance drops when training
on heterogeneous data. Recently, dataset distillation has been explored in
order to improve the efficiency and scalability of FL by creating a smaller,
synthetic dataset that retains the performance of a model trained on the local
private datasets. We discover that using distilled local datasets can amplify
the heterogeneity issue in FL. To address this, we propose a new method, called
Federated Virtual Learning on Heterogeneous Data with Local-Global Distillation
(FEDLGD), which trains FL using a smaller synthetic dataset (referred as
virtual data) created through a combination of local and global distillation.
Specifically, to handle synchronization and class imbalance, we propose
iterative distribution matching to allow clients to have the same amount of
balanced local virtual data; to harmonize the domain shifts, we use federated
gradient matching to distill global virtual data that are shared with clients
without hindering data privacy to rectify heterogeneous local training via
enforcing local-global feature similarity. We experiment on both benchmark and
real-world datasets that contain heterogeneous data from different sources. Our
method outperforms state-of-the-art heterogeneous FL algorithms under the
setting with a very limited amount of distilled virtual data.

本文提出了一种新的联邦虚拟学习方法，名为 Federated Virtual Learning on Heterogeneous Data with Local-Global Distillation (FEDLGD)，通过使用本地和全局蒸馏创建一个更小的合成数据集，以训练联邦学习，并解决了数据异构性带来的问题。实验表明，该方法在具有非常有限数量的精制虚拟数据的情况下，优于当前领先的异构 FL 算法。