Federated Learning (FL) seeks to train a model collaboratively without
sharing private training data from individual clients. Despite its promise, FL
encounters challenges such as high communication costs for large-scale models
and the necessity for uniform model architectures across all clients and the
server. These challenges severely restrict the practical applications of FL. To
address these limitations, the integration of knowledge distillation (KD) into
FL has been proposed, forming what is known as Federated Distillation (FD). FD
enables more flexible knowledge transfer between clients and the server,
surpassing the mere sharing of model parameters. By eliminating the need for
identical model architectures across clients and the server, FD mitigates the
communication costs associated with training large-scale models. This paper
aims to offer a comprehensive overview of FD, highlighting its latest
advancements. It delves into the fundamental principles underlying the design
of FD frameworks, delineates FD approaches for tackling various challenges, and
provides insights into the diverse applications of FD across different
scenarios.

该论文就联邦蒸馏（FD）提供了全面的概述，介绍了其最新进展，探讨了 FD 框架的基本原理，阐述了应对各种挑战的 FD 方法，并提供了 FD 在不同场景中的多样化应用。