We demonstrate the possibility of what we call sparse learning: accelerated
training of deep neural networks that maintain sparse weights throughout
training while achieving dense performance levels. We accomplish this by
developing sparse momentum, an algorithm which uses exponentially smoothed
gradients (momentum) to identify layers and weights which reduce the error
efficiently. Sparse momentum redistributes pruned weights across layers
according to the mean momentum magnitude of each layer. Within a layer, sparse
momentum grows weights according to the momentum magnitude of zero-valued
weights. We demonstrate state-of-the-art sparse performance on MNIST, CIFAR-10,
and ImageNet, decreasing the mean error by a relative 8%, 15%, and 6% compared
to other sparse algorithms. Furthermore, we show that sparse momentum reliably
reproduces dense performance levels while providing up to 5.61x faster
training. In our analysis, ablations show that the benefits of momentum
redistribution and growth increase with the depth and size of the network.
Additionally, we find that sparse momentum is insensitive to the choice of its
hyperparameters suggesting that sparse momentum is robust and easy to use.

本文研究了通过发展一种称为稀疏动量的算法，实现在深度神经网络训练过程中，保持稀疏权重的同时实现稠密表现水平的加速训练方法，实验证明稀疏动量可靠地重现稠密表现水平并提供最多 5.61 倍的训练加速度。