The reliance of language model training on massive amounts of computation and
vast datasets scraped from potentially low-quality, copyrighted, or sensitive
data has come into question practically, legally, and ethically. Federated
learning provides a plausible alternative by enabling previously untapped data
to be voluntarily gathered from collaborating organizations. However, when
scaled globally, federated learning requires collaboration across heterogeneous
legal, security, and privacy regimes while accounting for the inherent locality
of language data; this further exacerbates the established challenge of
federated statistical heterogeneity. We propose a Worldwide Federated Language
Model Training~(WorldLM) system based on federations of federations, where each
federation has the autonomy to account for factors such as its industry,
operating jurisdiction, or competitive environment. WorldLM enables such
autonomy in the presence of statistical heterogeneity via partial model
localization by allowing sub-federations to attentively aggregate key layers
from their constituents. Furthermore, it can adaptively share information
across federations via residual layer embeddings. Evaluations of language
modeling on naturally heterogeneous datasets show that WorldLM outperforms
standard federations by up to $1.91\times$, approaches the personalized
performance of fully local models, and maintains these advantages under
privacy-enhancing techniques.

通过联邦学习，在全球范围内构建联邦的语言模型训练系统 (WorldLM)，以解决大规模语言模型训练所面临的法律、安全、隐私等挑战，并通过局部模型本地化、残差层嵌入等方法，提升性能并保护隐私。

全球联合训练语言模型

Worldwide Federated Training of Language Models

To address increasing societal concerns regarding privacy and climate, the EU
adopted the General Data Protection Regulation (GDPR) and committed to the
Green Deal. Considerable research studied the energy efficiency of software and
the accuracy of machine learning models trained on anonymised data sets. Recent
work began exploring the impact of privacy-enhancing techniques (PET) on both
the energy consumption and accuracy of the machine learning models, focusing on
k-anonymity. As synthetic data is becoming an increasingly popular PET, this
paper analyses the energy consumption and accuracy of two phases: a) applying
privacy-enhancing techniques to the concerned data set, b) training the models
on the concerned privacy-enhanced data set. We use two privacy-enhancing
techniques: k-anonymisation (using generalisation and suppression) and
synthetic data, and three machine-learning models. Each model is trained on
each privacy-enhanced data set. Our results show that models trained on
k-anonymised data consume less energy than models trained on the original data,
with a similar performance regarding accuracy. Models trained on synthetic data
have a similar energy consumption and a similar to lower accuracy compared to
models trained on the original data.

本文研究了 k - 匿名、合成数据等隐私增强技术对机器学习模型能耗和准确性的影响，结果表明应用 k - 匿名到数据集可以降低能耗并保持准确度，而使用合成数据则可以保持能耗不变，但准确性可能会有所下降。