Network pruning is a promising avenue for compressing deep neural networks. A
typical approach to pruning starts by training a model and then removing
redundant parameters while minimizing the impact on what is learned.
Alternatively, a recent approach shows that pruning can be done at
initialization prior to training, based on a saliency criterion called
connection sensitivity. However, it remains unclear exactly why pruning an
untrained, randomly initialized neural network is effective. In this work, by
noting connection sensitivity as a form of gradient, we formally characterize
initialization conditions to ensure reliable connection sensitivity
measurements, which in turn yields effective pruning results. Moreover, we
analyze the signal propagation properties of the resulting pruned networks and
introduce a simple, data-free method to improve their trainability. Our
modifications to the existing pruning at initialization method lead to improved
results on all tested network models for image classification tasks.
Furthermore, we empirically study the effect of supervision for pruning and
demonstrate that our signal propagation perspective, combined with unsupervised
pruning, can be useful in various scenarios where pruning is applied to
non-standard arbitrarily-designed architectures.

该研究使用一种基于梯度的敏感度准则，阐述了初始化条件和网络结构剪枝的之间的关系，提出一种无需数据的改进方法，可用于图像分类任务中的先剪枝训练。

基于信号传播视角的神经网络初始化剪枝

A Signal Propagation Perspective for Pruning Neural Networks at  Initialization

Pruning large neural networks while maintaining their performance is often
desirable due to the reduced space and time complexity. In existing methods,
pruning is done within an iterative optimization procedure with either
heuristically designed pruning schedules or additional hyperparameters,
undermining their utility. In this work, we present a new approach that prunes
a given network once at initialization prior to training. To achieve this, we
introduce a saliency criterion based on connection sensitivity that identifies
structurally important connections in the network for the given task. This
eliminates the need for both pretraining and the complex pruning schedule while
making it robust to architecture variations. After pruning, the sparse network
is trained in the standard way. Our method obtains extremely sparse networks
with virtually the same accuracy as the reference network on the MNIST,
CIFAR-10, and Tiny-ImageNet classification tasks and is broadly applicable to
various architectures including convolutional, residual and recurrent networks.
Unlike existing methods, our approach enables us to demonstrate that the
retained connections are indeed relevant to the given task.

本文介绍了一种通过基于连接灵敏度的显著性标准在初始化前修剪神经网络，从而在保持网络性能的同时减少空间和时间复杂度的新方法，并应用于 MNIST、CIFAR-10 和 Tiny-ImageNet 分类任务的各种体系结构中，所得到的极度稀疏网络的准确性与基准网络几乎相同，而且能够证明保留的连接确实与给定任务相关。