We train language models (LMs) with federated learning (FL) and differential
privacy (DP) in the Google Keyboard (Gboard). We apply the
DP-Follow-the-Regularized-Leader (DP-FTRL)~\citep{kairouz21b} algorithm to
achieve meaningfully formal DP guarantees without requiring uniform sampling of
client devices. To provide favorable privacy-utility trade-offs, we introduce a
new client participation criterion and discuss the implication of its
configuration in large scale systems. We show how quantile-based clip
estimation~\citep{andrew2019differentially} can be combined with DP-FTRL to
adaptively choose the clip norm during training or reduce the hyperparameter
tuning in preparation for training. With the help of pretraining on public
data, we train and deploy more than twenty Gboard LMs that achieve high utility
and $\rho-$zCDP privacy guarantees with $\rho \in (0.2, 2)$, with two models
additionally trained with secure aggregation~\citep{bonawitz2017practical}. We
are happy to announce that all the next word prediction neural network LMs in
Gboard now have DP guarantees, and all future launches of Gboard neural network
LMs will require DP guarantees. We summarize our experience and provide
concrete suggestions on DP training for practitioners.

应用差分隐私和联邦学习算法，在 Google 键盘（Gboard）的多个模型中实现了高效且可靠的隐私保障，并提供了基于分位数估计的剪辑规范自适应方法，为学习者提供了具体建议。