Recent works on sparse neural network training (sparse training) have shown
that a compelling trade-off between performance and efficiency can be achieved
by training intrinsically sparse neural networks from scratch. Existing sparse
training methods usually strive to find the best sparse subnetwork possible in
one single run, without involving any expensive dense or pre-training steps.
For instance, dynamic sparse training (DST), is capable of reaching a
competitive performance of dense training by iteratively evolving the sparse
topology during the course of training. In this paper, we argue that it is
better to allocate the limited resources to create multiple low-loss sparse
subnetworks and superpose them into a stronger one, instead of allocating all
resources entirely to find an individual subnetwork. To achieve this, two
desiderata are required: (1) efficiently producing many low-loss subnetworks,
the so-called cheap tickets, within one training process limited to the
standard training time used in dense training; (2) effectively superposing
these cheap tickets into one stronger subnetwork. To corroborate our
conjecture, we present a novel sparse training approach, termed Sup-tickets,
which can satisfy the above two desiderata concurrently in a single
sparse-to-sparse training process. Across various modern architectures on
CIFAR-10/100 and ImageNet, we show that Sup-tickets integrates seamlessly with
the existing sparse training methods and demonstrates consistent performance
improvement.

本文提出了一种用有限的训练资源制造多个性价比高的稀疏神经网络子网络，并将它们叠加在一起形成一个更加强大的子网络的新型稀疏训练方法 Sup-tickets，该方法可以同时满足在标准训练时间内有效地生成多个低损失子网络和有效地将这些低成本子网络叠加成一个更强的子网络的两个愿望。在 CIFAR-10/100 和 ImageNet 等各种现代架构上展示了 Sup-tickets 的一致性性能改进。