Many Neural Network Pruning approaches consist of several iterative training
and pruning steps, seemingly losing a significant amount of their performance
after pruning and then recovering it in the subsequent retraining phase. Recent
works of Renda et al. (2020) and Le & Hua (2021) demonstrate the significance
of the learning rate schedule during the retraining phase and propose specific
heuristics for choosing such a schedule for IMP (Han et al., 2015). We place
these findings in the context of the results of Li et al. (2020) regarding the
training of models within a fixed training budget and demonstrate that,
consequently, the retraining phase can be massively shortened using a simple
linear learning rate schedule. Improving on existing retraining approaches, we
additionally propose a method to adaptively select the initial value of the
linear schedule. Going a step further, we propose similarly imposing a budget
on the initial dense training phase and show that the resulting simple and
efficient method is capable of outperforming significantly more complex or
heavily parameterized state-of-the-art approaches that attempt to sparsify the
network during training. These findings not only advance our understanding of
the retraining phase, but more broadly question the belief that one should aim
to avoid the need for retraining and reduce the negative effects of 'hard'
pruning by incorporating the sparsification process into the standard training.

该研究探讨了神经网络剪枝的相关问题，包括学习率、训练预算、线性学习率表等，并通过提出一个简单而有效的方法来适应性地选择线性表的初始值，进而减少网络的训练时间和复杂度。