Many neural network pruning algorithms proceed in three steps: train the
network to completion, remove unwanted structure to compress the network, and
retrain the remaining structure to recover lost accuracy. The standard
retraining technique, fine-tuning, trains the unpruned weights from their final
trained values using a small fixed learning rate. In this paper, we compare
fine-tuning to alternative retraining techniques. Weight rewinding (as proposed
by Frankle et al., (2019)), rewinds unpruned weights to their values from
earlier in training and retrains them from there using the original training
schedule. Learning rate rewinding (which we propose) trains the unpruned
weights from their final values using the same learning rate schedule as weight
rewinding. Both rewinding techniques outperform fine-tuning, forming the basis
of a network-agnostic pruning algorithm that matches the accuracy and
compression ratios of several more network-specific state-of-the-art
techniques.

本文提出了两种新的训练技术（Weight Rewinding 和 Learning Rate Rewinding）并将它们与传统 Fine Tuning 技术进行了比较，实验结果表明使用这两种技术进行神经网络剪枝可以获得与其他先进技术相当的压缩比和准确率。