Federated Learning (FL) has attracted much interest due to the significant
advantages it brings to training deep neural network (DNN) models. However,
since communications and computation resources are limited, training DNN models
in FL systems face challenges such as elevated computational and communication
costs in complex tasks. Sparse training schemes gain increasing attention in
order to scale down the dimensionality of each client (i.e., node)
transmission. Specifically, sparsification with error correction methods is a
promising technique, where only important updates are sent to the parameter
server (PS) and the rest are accumulated locally. While error correction
methods have shown to achieve a significant sparsification level of the
client-to-PS message without harming convergence, pushing sparsity further
remains unresolved due to the staleness effect. In this paper, we propose a
novel algorithm, dubbed Federated Learning with Accumulated Regularized
Embeddings (FLARE), to overcome this challenge. FLARE presents a novel sparse
training approach via accumulated pulling of the updated models with
regularization on the embeddings in the FL process, providing a powerful
solution to the staleness effect, and pushing sparsity to an exceptional level.
The performance of FLARE is validated through extensive experiments on diverse
and complex models, achieving a remarkable sparsity level (10 times and more
beyond the current state-of-the-art) along with significantly improved
accuracy. Additionally, an open-source software package has been developed for
the benefit of researchers and developers in related fields.

本文介绍了一种名为 FLARE 的新算法，通过对嵌入在 FL 过程中的累积拉取更新模型进行正则化，提供了解决过期效应并将稀疏性推向异常水平的强大解决方案。FLARE 的性能通过广泛的实验验证，在不同且复杂的模型上实现了显著的稀疏水平（超过当前技术水平的 10 倍以上）以及明显改善的准确性。此外，还开发了一个开源软件包，以造福相关领域的研究人员和开发人员。