The success of iterative pruning methods in achieving state-of-the-art sparse
networks has largely been attributed to improved mask identification and an
implicit regularization induced by pruning. We challenge this hypothesis and
instead posit that their repeated cyclic training schedules enable improved
optimization. To verify this, we show that pruning at initialization is
significantly boosted by repeated cyclic training, even outperforming standard
iterative pruning methods. The dominant mechanism how this is achieved, as we
conjecture, can be attributed to a better exploration of the loss landscape
leading to a lower training loss. However, at high sparsity, repeated cyclic
training alone is not enough for competitive performance. A strong coupling
between learnt parameter initialization and mask seems to be required. Standard
methods obtain this coupling via expensive pruning-training iterations,
starting from a dense network. To achieve this with sparse training instead, we
propose SCULPT-ing, i.e., repeated cyclic training of any sparse mask followed
by a single pruning step to couple the parameters and the mask, which is able
to match the performance of state-of-the-art iterative pruning methods in the
high sparsity regime at reduced computational cost.

通过重复周期性训练，我们提出了 SCULPT-ing 方法，即通过稀疏掩膜的重复周期性训练，然后进行单次剪枝步骤以耦合参数和掩膜，从而在减少计算成本的同时，在高稀疏度条件下达到与最先进的迭代剪枝方法相匹配的性能。

循环稀疏训练：足够吗？

Cyclic Sparse Training: Is it Enough?

The existence of "lottery tickets" arXiv:1803.03635 at or near initialization
raises the tantalizing question of whether large models are necessary in deep
learning, or whether sparse networks can be quickly identified and trained
without ever training the dense models that contain them. However, efforts to
find these sparse subnetworks without training the dense model ("pruning at
initialization") have been broadly unsuccessful arXiv:2009.08576. We put
forward a theoretical explanation for this, based on the model's effective
parameter count, $p_\text{eff}$, given by the sum of the number of non-zero
weights in the final network and the mutual information between the sparsity
mask and the data. We show the Law of Robustness of arXiv:2105.12806 extends to
sparse networks with the usual parameter count replaced by $p_\text{eff}$,
meaning a sparse neural network which robustly interpolates noisy data requires
a heavily data-dependent mask. We posit that pruning during and after training
outputs masks with higher mutual information than those produced by pruning at
initialization. Thus two networks may have the same sparsities, but differ in
effective parameter count based on how they were trained. This suggests that
pruning near initialization may be infeasible and explains why lottery tickets
exist, but cannot be found fast (i.e. without training the full network).
Experiments on neural networks confirm that information gained during training
may indeed affect model capacity.

彩票模型的存在考虑了深度学习中是否需要大型模型以及是否可以快速识别和训练稀疏网络，而无需训练包含它们的稠密模型。通过对彩票模型的理论解释，揭示了稀疏网络需要依赖于数据的遮罩来稳定插值噪声数据。研究证实了训练过程中获取的信息可以影响模型容量。

初始化时剪枝的信息论障碍

No Free Prune: Information-Theoretic Barriers to Pruning at  Initialization

Neural network pruning typically removes connections or neurons from a
pretrained converged model; while a new pruning paradigm, pruning at
initialization (PaI), attempts to prune a randomly initialized network. This
paper offers the first survey concentrated on this emerging pruning fashion. We
first introduce a generic formulation of neural network pruning, followed by
the major classic pruning topics. Then, as the main body of this paper, a
thorough and structured literature review of PaI methods is presented,
consisting of two major tracks (sparse training and sparse selection). Finally,
we summarize the surge of PaI compared to PaT and discuss the open problems.
Apart from the dedicated literature review, this paper also offers a code base
for easy sanity-checking and benchmarking of different PaI methods.

这篇论文首次调查了一种新兴的神经网络裁剪方式 —— 在初始化时裁剪（PaI），并介绍了其稀疏训练和稀疏选择两大主要方法。此外，该论文还提供了一个用于不同 PaI 方法的基准测试和检查的代码库。