Network pruning is an effective approach to reduce network complexity with
acceptable performance compromise. Existing studies achieve the sparsity of
neural networks via time-consuming weight training or complex searching on
networks with expanded width, which greatly limits the applications of network
pruning. In this paper, we show that high-performing and sparse sub-networks
without the involvement of weight training, termed "lottery jackpots", exist in
pre-trained models with unexpanded width. Furthermore, we improve the
efficiency for searching lottery jackpots from two perspectives. Firstly, we
observe that the sparse masks derived from many existing pruning criteria have
a high overlap with the searched mask of our lottery jackpot, among which, the
magnitude-based pruning results in the most similar mask with ours.
Consequently, our searched lottery jackpot removes 90% weights in ResNet-50,
while it easily obtains more than 70% top-1 accuracy using only 5 searching
epochs on ImageNet. In compliance with this insight, we initialize our sparse
mask using the magnitude-based pruning, resulting in at least 3x cost reduction
on the lottery jackpot searching while achieving comparable or even better
performance. Secondly, we conduct an in-depth analysis of the searching process
for lottery jackpots. Our theoretical result suggests that the decrease in
training loss during weight searching can be disturbed by the dependency
between weights in modern networks. To mitigate this, we propose a novel short
restriction method to restrict change of masks that may have potential negative
impacts on the training loss. Our code is available at
this https URL

本文提出了一种搜索被称为 "lottery jackpots" 的高性能、稀疏的子网络的方法，通过使用基于幅值的裁剪技术初始化稀疏掩码并使用短限制方法减少训练损失，从而提高搜索效率和准确性。