Sparse neural networks are becoming increasingly important as the field seeks
to improve the performance of existing models by scaling them up, while
simultaneously trying to reduce power consumption and computational footprint.
Unfortunately, most existing methods for inducing performant sparse models
still entail the instantiation of dense parameters, or dense gradients in the
backward-pass, during training. For very large models this requirement can be
prohibitive. In this work we propose Top-KAST, a method that preserves constant
sparsity throughout training (in both the forward and backward-passes). We
demonstrate the efficacy of our approach by showing that it performs comparably
to or better than previous works when training models on the established
ImageNet benchmark, whilst fully maintaining sparsity. In addition to our
ImageNet results, we also demonstrate our approach in the domain of language
modeling where the current best performing architectures tend to have tens of
billions of parameters and scaling up does not yet seem to have saturated
performance. Sparse versions of these architectures can be run with
significantly fewer resources, making them more widely accessible and
applicable. Furthermore, in addition to being effective, our approach is
straightforward and can easily be implemented in a wide range of existing
machine learning frameworks with only a few additional lines of code. We
therefore hope that our contribution will help enable the broader community to
explore the potential held by massive models, without incurring massive
computational cost.

本文提出了一种名为 Top-KAST 的方法，在训练时保持网络的稀疏性，避免了传统方法在训练时必须实例化 dense 参数或梯度的缺点，实验结果表明，该方法在 ImageNet 基准测试中表现出色，在语言建模领域也有广阔的应用前景。此外，该方法实现简单，易于在现有的机器学习框架中实现，为探索海量模型的潜力提供了可能。