The reliance of language model training on massive amounts of computation and
vast datasets scraped from potentially low-quality, copyrighted, or sensitive
data has come into question practically, legally, and ethically. Federated
learning provides a plausible alternative by enabling previously untapped data
to be voluntarily gathered from collaborating organizations. However, when
scaled globally, federated learning requires collaboration across heterogeneous
legal, security, and privacy regimes while accounting for the inherent locality
of language data; this further exacerbates the established challenge of
federated statistical heterogeneity. We propose a Worldwide Federated Language
Model Training~(WorldLM) system based on federations of federations, where each
federation has the autonomy to account for factors such as its industry,
operating jurisdiction, or competitive environment. WorldLM enables such
autonomy in the presence of statistical heterogeneity via partial model
localization by allowing sub-federations to attentively aggregate key layers
from their constituents. Furthermore, it can adaptively share information
across federations via residual layer embeddings. Evaluations of language
modeling on naturally heterogeneous datasets show that WorldLM outperforms
standard federations by up to $1.91\times$, approaches the personalized
performance of fully local models, and maintains these advantages under
privacy-enhancing techniques.

通过联邦学习，在全球范围内构建联邦的语言模型训练系统 (WorldLM)，以解决大规模语言模型训练所面临的法律、安全、隐私等挑战，并通过局部模型本地化、残差层嵌入等方法，提升性能并保护隐私。