The success of iterative pruning methods in achieving state-of-the-art sparse
networks has largely been attributed to improved mask identification and an
implicit regularization induced by pruning. We challenge this hypothesis and
instead posit that their repeated cyclic training schedules enable improved
optimization. To verify this, we show that pruning at initialization is
significantly boosted by repeated cyclic training, even outperforming standard
iterative pruning methods. The dominant mechanism how this is achieved, as we
conjecture, can be attributed to a better exploration of the loss landscape
leading to a lower training loss. However, at high sparsity, repeated cyclic
training alone is not enough for competitive performance. A strong coupling
between learnt parameter initialization and mask seems to be required. Standard
methods obtain this coupling via expensive pruning-training iterations,
starting from a dense network. To achieve this with sparse training instead, we
propose SCULPT-ing, i.e., repeated cyclic training of any sparse mask followed
by a single pruning step to couple the parameters and the mask, which is able
to match the performance of state-of-the-art iterative pruning methods in the
high sparsity regime at reduced computational cost.

通过重复周期性训练，我们提出了 SCULPT-ing 方法，即通过稀疏掩膜的重复周期性训练，然后进行单次剪枝步骤以耦合参数和掩膜，从而在减少计算成本的同时，在高稀疏度条件下达到与最先进的迭代剪枝方法相匹配的性能。

循环稀疏训练：足够吗？

Cyclic Sparse Training: Is it Enough?

Deciding the amount of neurons during the design of a deep neural network to
maximize performance is not intuitive. In this work, we attempt to search for
the neuron (filter) configuration of a fixed network architecture that
maximizes accuracy. Using iterative pruning methods as a proxy, we parameterize
the change of the neuron (filter) number of each layer with respect to the
change in parameters, allowing us to efficiently scale an architecture across
arbitrary sizes. We also introduce architecture descent which iteratively
refines the parameterized function used for model scaling. The combination of
both proposed methods is coined as NeuralScale. To prove the efficiency of
NeuralScale in terms of parameters, we show empirical simulations on VGG11,
MobileNetV2 and ResNet18 using CIFAR10, CIFAR100 and TinyImageNet as benchmark
datasets. Our results show an increase in accuracy of 3.04%, 8.56% and 3.41%
for VGG11, MobileNetV2 and ResNet18 on CIFAR10, CIFAR100 and TinyImageNet
respectively under a parameter-constrained setting (output neurons (filters) of
default configuration with scaling factor of 0.25).

本文提出了一种名为 NeuralScale 的算法，该算法使用了迭代剪枝方法控制神经网络的大小和参数，进而优化了固定神经网络的结构和配置，经过实验得出，NeuralScale 在参数约束的情况下，与默认配置相比，在 CIFAR10、CIFAR100 和 TinyImageNet 的数据集上分别增加了 3.04％，8.56％和 3.41％的准确性，取得了优异的表现。