Sparse neural networks have been widely applied to reduce the computational
demands of training and deploying over-parameterized deep neural networks. For
inference acceleration, methods that discover a sparse network from a
pre-trained dense network (dense-to-sparse training) work effectively.
Recently, dynamic sparse training (DST) has been proposed to train sparse
neural networks without pre-training a dense model (sparse-to-sparse training),
so that the training process can also be accelerated. However, previous
sparse-to-sparse methods mainly focus on Multilayer Perceptron Networks (MLPs)
and Convolutional Neural Networks (CNNs), failing to match the performance of
dense-to-sparse methods in the Recurrent Neural Networks (RNNs) setting. In
this paper, we propose an approach to train intrinsically sparse RNNs with a
fixed parameter count in one single run, without compromising performance.
During training, we allow RNN layers to have a non-uniform redistribution
across cell gates for better regularization. Further, we propose SNT-ASGD, a
novel variant of the averaged stochastic gradient optimizer, which
significantly improves the performance of all sparse training methods for RNNs.
Using these strategies, we achieve state-of-the-art sparse training results,
better than the dense-to-sparse methods, with various types of RNNs on Penn
TreeBank and Wikitext-2 datasets. Our codes are available at
this https URL

本文提出了一种可以在单次训练中固定参数数量的内在稀疏 RNNs 训练方法，利用非均匀分配细胞门的方法实现更好的正则化，并通过一种新型的平均随机梯度优化器 SNT-ASGD 提高了训练性能，在 Penn TreeBank 和 Wikitext-2 数据集上实现了优于 dense-to-sparse 方法的最新稀疏训练结果。