Mobile keyboard suggestion is typically regarded as a word-level language
modeling problem. Centralized machine learning technique requires massive user
data collected to train on, which may impose privacy concerns for sensitive
personal typing data of users. Federated learning (FL) provides a promising
approach to learning private language modeling for intelligent personalized
keyboard suggestion by training models in distributed clients rather than
training in a central server. To obtain a global model for prediction, existing
FL algorithms simply average the client models and ignore the importance of
each client during model aggregation. Furthermore, there is no optimization for
learning a well-generalized global model on the central server. To solve these
problems, we propose a novel model aggregation with the attention mechanism
considering the contribution of clients models to the global model, together
with an optimization technique during server aggregation. Our proposed
attentive aggregation method minimizes the weighted distance between the server
model and client models through iterative parameters updating while attends the
distance between the server model and client models. Through experiments on two
popular language modeling datasets and a social media dataset, our proposed
method outperforms its counterparts in terms of perplexity and communication
cost in most settings of comparison.

该研究提出了一种采用注意机制的模型聚合方法来解决传统联邦学习方法中模型聚合忽略客户端重要性问题的方法，并在两个流行的语言模型数据集和一个社交媒体数据集上进行了实验证明，在大部分比较中，该方法在模糊度和通信成本方面都优于传统方法。