Federated Learning (FL) is an emerging machine learning paradigm that enables
the collaborative training of a shared global model across distributed clients
while keeping the data decentralized. Recent works on designing systems for
efficient FL have shown that utilizing serverless computing technologies,
particularly Function-as-a-Service (FaaS) for FL, can enhance resource
efficiency, reduce training costs, and alleviate the complex infrastructure
management burden on data holders. However, existing serverless FL systems
implicitly assume a uniform global model architecture across all participating
clients during training. This assumption fails to address fundamental
challenges in practical FL due to the resource and statistical data
heterogeneity among FL clients. To address these challenges and enable
heterogeneous client models in serverless FL, we utilize Knowledge Distillation
(KD) in this paper. Towards this, we propose novel optimized serverless
workflows for two popular conventional federated KD techniques, i.e., FedMD and
FedDF. We implement these workflows by introducing several extensions to an
open-source serverless FL system called FedLess. Moreover, we comprehensively
evaluate the two strategies on multiple datasets across varying levels of
client data heterogeneity using heterogeneous client models with respect to
accuracy, fine-grained training times, and costs. Results from our experiments
demonstrate that serverless FedDF is more robust to extreme non-IID data
distributions, is faster, and leads to lower costs than serverless FedMD. In
addition, compared to the original implementation, our optimizations for
particular steps in FedMD and FedDF lead to an average speedup of 3.5x and
1.76x across all datasets.

利用优化的无服务器工作流程，在 FaaS 环境下实现基于知识蒸馏的异构客户端模型的联邦学习（FL）系统，实验证明无服务器 FedDF 比无服务器 FedMD 更快、成本更低且对极端非独立同分布数据分布更加鲁棒。

使用知识蒸馏在无服务器联邦学习中训练异构客户端模型

Training Heterogeneous Client Models using Knowledge Distillation in  Serverless Federated Learning

Federated Learning (FL) is a machine learning setting where many devices
collaboratively train a machine learning model while keeping the training data
decentralized. In most of the current training schemes the central model is
refined by averaging the parameters of the server model and the updated
parameters from the client side. However, directly averaging model parameters
is only possible if all models have the same structure and size, which could be
a restrictive constraint in many scenarios.
In this work we investigate more powerful and more flexible aggregation
schemes for FL. Specifically, we propose ensemble distillation for model
fusion, i.e. training the central classifier through unlabeled data on the
outputs of the models from the clients. This knowledge distillation technique
mitigates privacy risk and cost to the same extent as the baseline FL
algorithms, but allows flexible aggregation over heterogeneous client models
that can differ e.g. in size, numerical precision or structure. We show in
extensive empirical experiments on various CV/NLP datasets (CIFAR-10/100,
ImageNet, AG News, SST2) and settings (heterogeneous models/data) that the
server model can be trained much faster, requiring fewer communication rounds
than any existing FL technique so far.

本篇论文研究了 FL 领域中不同于普通平均策略的模型聚合方法 ——ensemble distillation，证明其能够在保证数据隐私性和节约算力的同时，更灵活地聚合异构模型，并在各种 CV/NLP 数据集和场景下优于现有的 FL 算法。