Works on lottery ticket hypothesis (LTH) and single-shot network pruning
(SNIP) have raised a lot of attention currently on post-training pruning
(iterative magnitude pruning), and before-training pruning (pruning at
initialization). The former method suffers from an extremely large computation
cost and the latter usually struggles with insufficient performance. In
comparison, during-training pruning, a class of pruning methods that
simultaneously enjoys the training/inference efficiency and the comparable
performance, temporarily, has been less explored. To better understand
during-training pruning, we quantitatively study the effect of pruning
throughout training from the perspective of pruning plasticity (the ability of
the pruned networks to recover the original performance). Pruning plasticity
can help explain several other empirical observations about neural network
pruning in literature. We further find that pruning plasticity can be
substantially improved by injecting a brain-inspired mechanism called
neuroregeneration, i.e., to regenerate the same number of connections as
pruned. We design a novel gradual magnitude pruning (GMP) method, named gradual
pruning with zero-cost neuroregeneration (\textbf{GraNet}), that advances state
of the art. Perhaps most impressively, its sparse-to-sparse version for the
first time boosts the sparse-to-sparse training performance over various
dense-to-sparse methods with ResNet-50 on ImageNet without extending the
training time. We release all codes in
this https URL

本论文研究了通过神经再生技术改进的渐进式剪枝算法（GraNet），它结合了取得与单次剪枝同等性能和训练 / 推理效率的优点，尤其是其稀疏 - 稀疏模式极大地提高了 ResNet-50 在 ImageNet 上的性能。