A number of recent approaches have been proposed for pruning neural network
parameters at initialization with the goal of reducing the size and
computational burden of models while minimally affecting their training
dynamics and generalization performance. While each of these approaches have
some amount of well-founded motivation, a rigorous analysis of the effect of
these pruning methods on network training dynamics and their formal
relationship to each other has thus far received little attention. Leveraging
recent theoretical approximations provided by the Neural Tangent Kernel, we
unify a number of popular approaches for pruning at initialization under a
single path-centric framework. We introduce the Path Kernel as the
data-independent factor in a decomposition of the Neural Tangent Kernel and
show the global structure of the Path Kernel can be computed efficiently. This
Path Kernel decomposition separates the architectural effects from the
data-dependent effects within the Neural Tangent Kernel, providing a means to
predict the convergence dynamics of a network from its architecture alone. We
analyze the use of this structure in approximating training and generalization
performance of networks in the absence of data across a number of
initialization pruning approaches. Observing the relationship between input
data and paths and the relationship between the Path Kernel and its natural
norm, we additionally propose two augmentations of the SynFlow algorithm for
pruning at initialization.

使用神经切向核的理论近似，通过引入路径核将几种网络剪枝方法统一到单一的路径中心框架下，并分析了其对网络训练动态的影响及其相互关系，进而提出了两种初始化剪枝算法的改进方法。

初始化剪枝的统一路径视角

A Unified Paths Perspective for Pruning at Initialization

Recent work has explored the possibility of pruning neural networks at
initialization. We assess proposals for doing so: SNIP (Lee et al., 2019),
GraSP (Wang et al., 2020), SynFlow (Tanaka et al., 2020), and magnitude
pruning. Although these methods surpass the trivial baseline of random pruning,
they remain below the accuracy of magnitude pruning after training, and we
endeavor to understand why. We show that, unlike pruning after training,
randomly shuffling the weights these methods prune within each layer or
sampling new initial values preserves or improves accuracy. As such, the
per-weight pruning decisions made by these methods can be replaced by a
per-layer choice of the fraction of weights to prune. This property suggests
broader challenges with the underlying pruning heuristics, the desire to prune
at initialization, or both.

研究表明神经网络在初始化阶段进行剪枝是可能的，但是现有方法（SNIP、GraSP、SynFlow 和 magnitude pruning）表现不如训练后的 magnitude pruning，可能是因为这些方法的权重剪枝决策可以通过每层选择剪枝权重比例来替换，这反映出底层剪枝启发式算法及初始化阶段进行剪枝的挑战。