Recent advances in neural network pruning have shown how it is possible to
reduce the computational costs and memory demands of deep learning models
before training. We focus on this framework and propose a new pruning at
initialization algorithm that leverages the Neural Tangent Kernel (NTK) theory
to align the training dynamics of the sparse network with that of the dense
one. Specifically, we show how the usually neglected data-dependent component
in the NTK's spectrum can be taken into account by providing an analytical
upper bound to the NTK's trace obtained by decomposing neural networks into
individual paths. This leads to our Path eXclusion (PX), a foresight pruning
method designed to preserve the parameters that mostly influence the NTK's
trace. PX is able to find lottery tickets (i.e. good paths) even at high
sparsity levels and largely reduces the need for additional training. When
applied to pre-trained models it extracts subnetworks directly usable for
several downstream tasks, resulting in performance comparable to those of the
dense counterpart but with substantial cost and computational savings. Code
available at: this https URL

神经网络修剪能够减少深度学习模型训练之前的计算成本和内存需求。我们专注于该框架，并提出了一种新的初始化修剪算法，利用神经切向核（NTK）理论将稀疏网络的训练动态与密集网络对齐。我们介绍了如何考虑 NTK 光谱中通常被忽视的数据相关部分，并通过将神经网络分解为单个路径来获得 NTK 迹的解析上界。这导致我们的 Path eXclusion (PX) 方法，一种旨在保留对 NTK 迹影响最大的参数的预测性修剪方法。PX 能够在高稀疏程度下找到优秀的路径，大大减少额外的训练需求。应用于预训练模型时，它提取出可用于多个下游任务的子网络，性能可与密集模型媲美，但具有大量成本和计算节约。