This study develops a federated learning (FL) framework overcoming largely
incremental communication costs due to model sizes in typical frameworks
without compromising model performance. To this end, based on the idea of
leveraging an unlabeled open dataset, we propose a distillation-based
semi-supervised FL (DS-FL) algorithm that exchanges the outputs of local models
among mobile devices, instead of model parameter exchange employed by the
typical frameworks. In DS-FL, the communication cost depends only on the output
dimensions of the models and does not scale up according to the model size. The
exchanged model outputs are used to label each sample of the open dataset,
which creates an additionally labeled dataset. Based on the new dataset, local
models are further trained, and model performance is enhanced owing to the data
augmentation effect. We further highlight that in DS-FL, the heterogeneity of
the devices' dataset leads to ambiguous of each data sample and lowing of the
training convergence. To prevent this, we propose entropy reduction averaging,
where the aggregated model outputs are intentionally sharpened. Moreover,
extensive experiments show that DS-FL reduces communication costs up to 99%
relative to those of the FL benchmark while achieving similar or higher
classification accuracy.

该研究提出了一种新的联邦学习框架，利用无标签的开放数据集和基于蒸馏的半监督学习算法，实现了模型的交流和性能的提高，并通过熵的减少平均值来减少模型之间的异质性。实验结果表明，相对于 FL 基准，DS-FL 能降低高达 99% 的通信成本，同时达到相似甚至更高的分类准确性。