Modern deep neural networks (DNNs) consist of millions of parameters,
necessitating high-performance computing during training and inference. Pruning
is one solution that significantly reduces the space and time complexities of
DNNs. Traditional pruning methods that are applied post-training focus on
streamlining inference, but there are recent efforts to leverage sparsity early
on by pruning before training. Pruning methods, such as iterative
magnitude-based pruning (IMP) achieve up to a 90% parameter reduction while
retaining accuracy comparable to the original model. However, this leads to
impractical runtime as it relies on multiple train-prune-reset cycles to
identify and eliminate redundant parameters. In contrast, training agnostic
early pruning methods, such as SNIP and SynFlow offer fast pruning but fall
short of the accuracy achieved by IMP at high sparsities. To bridge this gap,
we present Dual Gradient-Based Rapid Iterative Pruning (DRIVE), which leverages
dense training for initial epochs to counteract the randomness inherent at the
initialization. Subsequently, it employs a unique dual gradient-based metric
for parameter ranking. It has been experimentally demonstrated for VGG and
ResNet architectures on CIFAR-10/100 and Tiny ImageNet, and ResNet on ImageNet
that DRIVE consistently has superior performance over other training-agnostic
early pruning methods in accuracy. Notably, DRIVE is 43$\times$ to 869$\times$
faster than IMP for pruning.

现代深度神经网络 (DNNs) 由数百万个参数组成，在训练和推断过程中需要高性能计算。修剪是一个显著减少 DNNs 空间和时间复杂度的解决方案。

DRIVE: 双梯度快速迭代剪枝

DRIVE: Dual Gradient-Based Rapid Iterative Pruning

Lottery Ticket Hypothesis (LTH) raises keen attention to identifying sparse
trainable subnetworks, or winning tickets, which can be trained in isolation to
achieve similar or even better performance compared to the full models. Despite
many efforts being made, the most effective method to identify such winning
tickets is still Iterative Magnitude-based Pruning (IMP), which is
computationally expensive and has to be run thoroughly for every different
network. A natural question that comes in is: can we "transform" the winning
ticket found in one network to another with a different architecture, yielding
a winning ticket for the latter at the beginning, without re-doing the
expensive IMP? Answering this question is not only practically relevant for
efficient "once-for-all" winning ticket finding, but also theoretically
appealing for uncovering inherently scalable sparse patterns in networks. We
conduct extensive experiments on CIFAR-10 and ImageNet, and propose a variety
of strategies to tweak the winning tickets found from different networks of the
same model family (e.g., ResNets). Based on these results, we articulate the
Elastic Lottery Ticket Hypothesis (E-LTH): by mindfully replicating (or
dropping) and re-ordering layers for one network, its corresponding winning
ticket could be stretched (or squeezed) into a subnetwork for another deeper
(or shallower) network from the same family, whose performance is nearly the
same competitive as the latter's winning ticket directly found by IMP. We have
also extensively compared E-LTH with pruning-at-initialization and dynamic
sparse training methods, as well as discussed the generalizability of E-LTH to
different model families, layer types, and across datasets. Code is available
at this https URL

本文提出了一种弹性彩票假设（Elastic Lottery Ticket Hypothesis），证明了通过适当地调整一种深度学习神经网络的层级结构，可以从同一类神经网络的另一个较深或较浅网络中拉伸或压缩其获胜彩票，从而实现相当于 IMP 直接发现的性能。