In the recent paradigm of Federated Learning (FL), multiple clients train a
shared model while keeping their local data private. Resource constraints of
clients and communication costs pose major problems for training large models
in FL. On the one hand, addressing the resource limitations of the clients,
sparse training has proven to be a powerful tool in the centralized setting. On
the other hand, communication costs in FL can be addressed by local training,
where each client takes multiple gradient steps on its local data. Recent work
has shown that local training can provably achieve the optimal accelerated
communication complexity [Mishchenko et al., 2022]. Hence, one would like an
accelerated sparse training algorithm. In this work we show that naive
integration of sparse training and acceleration at the server fails, and how to
fix it by letting the clients perform these tasks appropriately. We introduce
Sparse-ProxSkip, our method developed for the nonconvex setting, inspired by
RandProx [Condat and Richt\'arik, 2022], which provably combines sparse
training and acceleration in the convex setting. We demonstrate the good
performance of Sparse-ProxSkip in extensive experiments.

该研究论文介绍了稀疏训练和加速通信在 Federated Learning 中的整合方法 Sparse-ProxSkip，并在大量实验证明了其良好性能。