Recent network pruning methods focus on pruning models early-on in training.
To estimate the impact of removing a parameter, these methods use importance
measures that were originally designed to prune trained models. Despite lacking
justification for their use early-on in training, such measures result in
surprisingly low accuracy loss. To better explain this behavior, we develop a
general framework that uses gradient flow to unify state-of-the-art importance
measures through the norm of model parameters. We use this framework to
determine the relationship between pruning measures and evolution of model
parameters, establishing several results related to pruning models early-on in
training: (i) magnitude-based pruning removes parameters that contribute least
to reduction in loss, resulting in models that converge faster than
magnitude-agnostic methods; (ii) loss-preservation based pruning preserves
first-order model evolution dynamics and is therefore appropriate for pruning
minimally trained models; and (iii) gradient-norm based pruning affects
second-order model evolution dynamics, such that increasing gradient norm via
pruning can produce poorly performing models. We validate our claims on several
VGG-13, MobileNet-V1, and ResNet-56 models trained on CIFAR-10/CIFAR-100. Code
available at this https URL

该篇研究通过梯度流的方式，研究了深度神经网络早期剪枝的影响，通过多组实验验证了几种不同的影响度量标准，从而在神经网络剪枝的早期阶段提出了一种更加有效的剪枝方法。