We propose algorithms to train production-quality n-gram language models
using federated learning. Federated learning is a distributed computation
platform that can be used to train global models for portable devices such as
smart phones. Federated learning is especially relevant for applications
handling privacy-sensitive data, such as virtual keyboards, because training is
performed without the users' data ever leaving their devices. While the
principles of federated learning are fairly generic, its methodology assumes
that the underlying models are neural networks. However, virtual keyboards are
typically powered by n-gram language models for latency reasons.
We propose to train a recurrent neural network language model using the
decentralized FederatedAveraging algorithm and to approximate this federated
model server-side with an n-gram model that can be deployed to devices for fast
inference. Our technical contributions include ways of handling large
vocabularies, algorithms to correct capitalization errors in user data, and
efficient finite state transducer algorithms to convert word language models to
word-piece language models and vice versa. The n-gram language models trained
with federated learning are compared to n-grams trained with traditional
server-based algorithms using A/B tests on tens of millions of users of virtual
keyboard. Results are presented for two languages, American English and
Brazilian Portuguese. This work demonstrates that high-quality n-gram language
models can be trained directly on client mobile devices without sensitive
training data ever leaving the devices.

提出使用联邦学习算法来训练生产级的 n-gram 语言模型，以解决在处理隐私敏感数据时面临的数据安全问题，该算法特别适用于便携式设备，如智能手机。同时通过将递归神经网络语言模型近似为能快速部署在设备上进行推理的 n-gram 模型，实现了联邦学习在虚拟键盘等场景下的应用。