The excessive computational requirements of modern artificial neural networks
(ANNs) are posing limitations on the machines that can run them. Sparsification
of ANNs is often motivated by time, memory and energy savings only during model
inference, yielding no benefits during training. A growing body of work is now
focusing on providing the benefits of model sparsification also during
training. While these methods greatly improve the training efficiency, the
training algorithms yielding the most accurate models still materialize the
dense weights, or compute dense gradients during training. We propose an
efficient, always-sparse training algorithm with excellent scaling to larger
and sparser models, supported by its linear time complexity with respect to the
model width during training and inference. Moreover, our guided stochastic
exploration algorithm improves over the accuracy of previous sparse training
methods. We evaluate our method on CIFAR-10/100 and ImageNet using ResNet, VGG,
and ViT models, and compare it against a range of sparsification methods.

现代人工神经网络的过多计算需求为可以运行它们的机器带来了限制。我们提出一种高效的、始终稀疏训练算法，具有一流的大规模和更稀疏模型的线性时间复杂度，并通过引导随机探索算法改善了先前稀疏训练方法的准确性。