Federated learning (FL) is a heavily promoted approach for training ML models on sensitive data, e.g., text typed by users on their smartphones. FL is expressly designed for training on data that are unbalanced and non-iid across the participants. To ensure privacy and integrity of the federated model, latest FL approaches use differential privacy or robust aggregation to limit the influence of "outlier" participants. First, we show that on standard tasks such as next-word prediction, many participants gain no benefit from FL because the federated model is less accurate on their data than the models they can train locally on their own. Second, we show that differential privacy and robust aggregation make this problem worse by further destroying the accuracy of the federated model for many participants. Then, we evaluate three techniques for local adaptation of federated models: fine-tuning, multi-task learning, and knowledge distillation. We analyze where each technique is applicable and demonstrate that all participants benefit from local adaptation. Participants whose local models are poor obtain big accuracy improvements over conventional FL. Participants whose local models are better than the federated model and who have no incentive to participate in FL today improve less, but sufficiently to make the adapted federated model better than their local models.

针对联邦学习中参与者局部利益与数据准确度不匹配的问题，本文提出并比较了三种本地适应技术，并指出差分隐私和鲁棒聚合恶化了联邦模型的准确性。这三种技术分别为：微调、多任务学习和知识蒸馏。我们的实验结果表明，所有参与者都从本地适应中受益，并且本地模型表现不佳的参与者通过传统联邦方式得到了大幅提升。

通过本地适配挽救联合学习