Sparse training is emerging as a promising avenue for reducing the
computational cost of training neural networks. Several recent studies have
proposed pruning methods using learnable thresholds to efficiently explore the
non-uniform distribution of sparsity inherent within the models. In this paper,
we propose Gradient Annealing (GA), where gradients of masked weights are
scaled down in a non-linear manner. GA provides an elegant trade-off between
sparsity and accuracy without the need for additional sparsity-inducing
regularization. We integrated GA with the latest learnable pruning methods to
create an automated sparse training algorithm called AutoSparse, which achieves
better accuracy and/or training/inference FLOPS reduction than existing
learnable pruning methods for sparse ResNet50 and MobileNetV1 on ImageNet-1K:
AutoSparse achieves (2x, 7x) reduction in (training,inference) FLOPS for
ResNet50 on ImageNet at 80% sparsity. Finally, AutoSparse outperforms
sparse-to-sparse SotA method MEST (uniform sparsity) for 80% sparse ResNet50
with similar accuracy, where MEST uses 12% more training FLOPS and 50% more
inference FLOPS.

本文提出通过渐进式变化策略的梯度退火（gradient annealing，GA）以及最新的可学习剪枝方法相结合的自动稀疏训练算法 AutoSparse，在 ImageNet-1K 数据集上表现优异，80% 稀疏 ResNet50 的训练和推断 FLOPS 减少分别达到 2 倍和 7 倍。与当前最好的稀疏到稀疏（sparse-to-sparse）方法 MEST 的表现相似，但使用的训练和推理 FLOPS 分别多 12％和 50％不到。