Federated learning (FL) is an important paradigm for training global models
from decentralized data in a privacy-preserving way. Existing FL methods
usually assume the global model can be trained on any participating client.
However, in real applications, the devices of clients are usually
heterogeneous, and have different computing power. Although big models like
BERT have achieved huge success in AI, it is difficult to apply them to
heterogeneous FL with weak clients. The straightforward solutions like removing
the weak clients or using a small model to fit all clients will lead to some
problems, such as under-representation of dropped clients and inferior accuracy
due to data loss or limited model representation ability. In this work, we
propose InclusiveFL, a client-inclusive federated learning method to handle
this problem. The core idea of InclusiveFL is to assign models of different
sizes to clients with different computing capabilities, bigger models for
powerful clients and smaller ones for weak clients. We also propose an
effective method to share the knowledge among multiple local models with
different sizes. In this way, all the clients can participate in the model
learning in FL, and the final model can be big and powerful enough. Besides, we
propose a momentum knowledge distillation method to better transfer knowledge
in big models on powerful clients to the small models on weak clients.
Extensive experiments on many real-world benchmark datasets demonstrate the
effectiveness of the proposed method in learning accurate models from clients
with heterogeneous devices under the FL framework.

该研究提出了一种名为 InclusiveFL 的客户端包含的联邦学习方法，解决异构设备下的全局模型训练问题，并通过在不同大小的本地模型之间共享知识和动量知识蒸馏方法来实现客户端之间的模型学习和知识迁移，实验表明该方法在联邦学习框架下从客户异构设备中学习准确模型方面具有很好的效果。