Federated Learning (FL) is a novel machine learning approach that allows the
model trainer to access more data samples, by training the model across
multiple decentralized data sources, while data access constraints are in
place. Such trained models can achieve significantly higher performance beyond
what can be done when trained on a single data source. As part of FL's
promises, none of the training data is ever transmitted to any central
location, ensuring that sensitive data remains local and private. These
characteristics make FL perfectly suited for large-scale applications in
healthcare, where a variety of compliance constraints restrict how data may be
handled, processed, and stored. Despite the apparent benefits of federated
learning, the heterogeneity in the local data distributions pose significant
challenges, and such challenges are even more pronounced in the case of
multilingual data providers. In this paper we present a federated learning
system for training a large-scale multi-lingual model suitable for fine-tuning
on downstream tasks such as medical entity tagging. Our work represents one of
the first such production-scale systems, capable of training across multiple
highly heterogeneous data providers, and achieving levels of accuracy that
could not be otherwise achieved by using central training with public data.
Finally, we show that the global model performance can be further improved by a
training step performed locally.

该论文介绍了一个针对医学实体标注等下游任务的大规模多语言模型的联邦学习系统，它能够跨越高度异质化的数据提供者进行训练，并通过本地训练步骤进一步提高全局模型的性能。