Federated learning (FL) enables edge-devices to collaboratively learn a model
without disclosing their private data to a central aggregating server. Most
existing FL algorithms require models of identical architecture to be deployed
across the clients and server, making it infeasible to train large models due
to clients' limited system resources. In this work, we propose a novel ensemble
knowledge transfer method named Fed-ET in which small models (different in
architecture) are trained on clients, and used to train a larger model at the
server. Unlike in conventional ensemble learning, in FL the ensemble can be
trained on clients' highly heterogeneous data. Cognizant of this property,
Fed-ET uses a weighted consensus distillation scheme with diversity
regularization that efficiently extracts reliable consensus from the ensemble
while improving generalization by exploiting the diversity within the ensemble.
We show the generalization bound for the ensemble of weighted models trained on
heterogeneous datasets that supports the intuition of Fed-ET. Our experiments
on image and language tasks show that Fed-ET significantly outperforms other
state-of-the-art FL algorithms with fewer communicated parameters, and is also
robust against high data-heterogeneity.

本研究提出了一种新的合奏知识转移方法 Fed-ET，采用了加权一致性蒸馏方案，通过利用各服务器的高度异构数据来提取可靠的共识，同时通过利用合奏中的多样性来改善泛化能力，并在图像和语言任务中显示出在通信参数较少的情况下，Fed-ET 明显优于其他最先进的 FL 算法，并且对于高数据异构性也是强健的。