The success of iterative pruning methods in achieving state-of-the-art sparse
networks has largely been attributed to improved mask identification and an
implicit regularization induced by pruning. We challenge this hypothesis and
instead posit that their repeated cyclic training schedules enable improved
optimization. To verify this, we show that pruning at initialization is
significantly boosted by repeated cyclic training, even outperforming standard
iterative pruning methods. The dominant mechanism how this is achieved, as we
conjecture, can be attributed to a better exploration of the loss landscape
leading to a lower training loss. However, at high sparsity, repeated cyclic
training alone is not enough for competitive performance. A strong coupling
between learnt parameter initialization and mask seems to be required. Standard
methods obtain this coupling via expensive pruning-training iterations,
starting from a dense network. To achieve this with sparse training instead, we
propose SCULPT-ing, i.e., repeated cyclic training of any sparse mask followed
by a single pruning step to couple the parameters and the mask, which is able
to match the performance of state-of-the-art iterative pruning methods in the
high sparsity regime at reduced computational cost.

通过重复周期性训练，我们提出了 SCULPT-ing 方法，即通过稀疏掩膜的重复周期性训练，然后进行单次剪枝步骤以耦合参数和掩膜，从而在减少计算成本的同时，在高稀疏度条件下达到与最先进的迭代剪枝方法相匹配的性能。

循环稀疏训练：足够吗？

Cyclic Sparse Training: Is it Enough?

This paper proposes a training method having multiple cyclic training for
achieving enhanced performance in low-bit quantized convolutional neural
networks (CNNs). Quantization is a popular method for obtaining lightweight
CNNs, where the initialization with a pretrained model is widely used to
overcome degraded performance in low-resolution quantization. However, large
quantization errors between real values and their low-bit quantized ones cause
difficulties in achieving acceptable performance for complex networks and large
datasets. The proposed training method softly delivers the knowledge of
pretrained models to low-bit quantized models in multiple quantization steps.
In each quantization step, the trained weights of a model are used to
initialize the weights of the next model with the quantization bit depth
reduced by one. With small change of the quantization bit depth, the
performance gap can be bridged, thus providing better weight initialization. In
cyclic training, after training a low-bit quantized model, its trained weights
are used in the initialization of its accurate model to be trained. By using
better training ability of the accurate model in an iterative manner, the
proposed method can produce enhanced trained weights for the low-bit quantized
model in each cycle. Notably, the training method can advance Top-1 and Top-5
accuracies of the binarized ResNet-18 on the ImageNet dataset by 5.80% and
6.85%, respectively.

本文提出了一种通过多次循环训练来实现强化低位量化卷积神经网络性能的训练方法，该方法通过使用多个量化步骤，软性地将预训练模型的知识传递给低位量化模型，并采用循环训练来提高模型的准确性，使得该方法能够在 ImageNet 数据集上提高 ResNet-18 二值化网络的 Top-1 和 Top-5 精度分别达到 5.80％和 6.85％。