Randomly initialized dense networks contain subnetworks that achieve high
accuracy without weight learning -- strong lottery tickets (SLTs). Recently,
Gadhikar et al. (2023) demonstrated theoretically and experimentally that SLTs
can also be found within a randomly pruned source network, thus reducing the
SLT search space. However, this limits the search to SLTs that are even sparser
than the source, leading to worse accuracy due to unintentionally high
sparsity. This paper proposes a method that reduces the SLT search space by an
arbitrary ratio that is independent of the desired SLT sparsity. A random
subset of the initial weights is excluded from the search space by freezing it
-- i.e., by either permanently pruning them or locking them as a fixed part of
the SLT. Indeed, the SLT existence in such a reduced search space is
theoretically guaranteed by our subset-sum approximation with randomly frozen
variables. In addition to reducing search space, the random freezing pattern
can also be exploited to reduce model size in inference. Furthermore,
experimental results show that the proposed method finds SLTs with better
accuracy and model size trade-off than the SLTs obtained from dense or randomly
pruned source networks. In particular, the SLT found in a frozen graph neural
network achieves higher accuracy than its weight trained counterpart while
reducing model size by $40.3\times$.

提出了一种方法，通过冻结一部分初始权重来减少 SLT（强彩票）的搜索空间，以在减小模型尺寸和提高精度之间取得平衡。实验证明，该方法能够找到更好的 SLTs，并且冻结图神经网络中的 SLT 能够在提高准确度的同时减小模型尺寸 40.3 倍。

在冻结网络中部分搜索足以找到一个强大的中奖彩票

Partial Search in a Frozen Network is Enough to Find a Strong Lottery  Ticket

Random pruning is arguably the most naive way to attain sparsity in neural
networks, but has been deemed uncompetitive by either post-training pruning or
sparse training. In this paper, we focus on sparse training and highlight a
perhaps counter-intuitive finding, that random pruning at initialization can be
quite powerful for the sparse training of modern neural networks. Without any
delicate pruning criteria or carefully pursued sparsity structures, we
empirically demonstrate that sparsely training a randomly pruned network from
scratch can match the performance of its dense equivalent. There are two key
factors that contribute to this revival: (i) the network sizes matter: as the
original dense networks grow wider and deeper, the performance of training a
randomly pruned sparse network will quickly grow to matching that of its dense
equivalent, even at high sparsity ratios; (ii) appropriate layer-wise sparsity
ratios can be pre-chosen for sparse training, which shows to be another
important performance booster. Simple as it looks, a randomly pruned subnetwork
of Wide ResNet-50 can be sparsely trained to outperforming a dense Wide
ResNet-50, on ImageNet. We also observed such randomly pruned networks
outperform dense counterparts in other favorable aspects, such as
out-of-distribution detection, uncertainty estimation, and adversarial
robustness. Overall, our results strongly suggest there is larger-than-expected
room for sparse training at scale, and the benefits of sparsity might be more
universal beyond carefully designed pruning. Our source code can be found at
this https URL.

本研究探讨了如何在现代神经网络中使用稀疏训练，提出了初始化时的随机剪枝能够有效地提高神经网络的稀疏训练性能，结果表明此方法可以匹配对应的密集网络，达到了预期效果，并且进行适当的层级稀疏比率的选择，可以进一步提高性能。