Many applications require sparse neural networks due to space or inference
time restrictions. There is a large body of work on training dense networks to
yield sparse networks for inference, but this limits the size of the largest
trainable sparse model to that of the largest trainable dense model. In this
paper we introduce a method to train sparse neural networks with a fixed
parameter count and a fixed computational cost throughout training, without
sacrificing accuracy relative to existing dense-to-sparse training methods. Our
method updates the topology of the sparse network during training by using
parameter magnitudes and infrequent gradient calculations. We show that this
approach requires fewer floating-point operations (FLOPs) to achieve a given
level of accuracy compared to prior techniques. We demonstrate state-of-the-art
sparse training results on a variety of networks and datasets, including
ResNet-50, MobileNets on Imagenet-2012, and RNNs on WikiText-103. Finally, we
provide some insights into why allowing the topology to change during the
optimization can overcome local minima encountered when the topology remains
static. Code used in our work can be found in github.com/google-research/rigl.

本论文提出了一种方法来训练稀疏神经网络，使其在训练过程中保持固定参数数量和固定的计算成本，同时不损失相对于现有密集到稀疏训练方法的准确性，该方法通过使用参数大小和不频繁的梯度计算来更新稀疏网络的拓扑结构。在多种网络和数据集上，我们展示了稀疏训练结果在 ResNet-50、MobileNets on Imagenet-2012、以及 RNNs on WikiText-103 方面的最新技术进展。最终，我们提供了一些见解，为什么允许在优化过程中更改稀疏神经网络拓扑结构可以克服当拓扑结构保持静态时遇到的局部极小值。