Lottery Ticket Hypothesis (LTH) raises keen attention to identifying sparse
trainable subnetworks, or winning tickets, which can be trained in isolation to
achieve similar or even better performance compared to the full models. Despite
many efforts being made, the most effective method to identify such winning
tickets is still Iterative Magnitude-based Pruning (IMP), which is
computationally expensive and has to be run thoroughly for every different
network. A natural question that comes in is: can we "transform" the winning
ticket found in one network to another with a different architecture, yielding
a winning ticket for the latter at the beginning, without re-doing the
expensive IMP? Answering this question is not only practically relevant for
efficient "once-for-all" winning ticket finding, but also theoretically
appealing for uncovering inherently scalable sparse patterns in networks. We
conduct extensive experiments on CIFAR-10 and ImageNet, and propose a variety
of strategies to tweak the winning tickets found from different networks of the
same model family (e.g., ResNets). Based on these results, we articulate the
Elastic Lottery Ticket Hypothesis (E-LTH): by mindfully replicating (or
dropping) and re-ordering layers for one network, its corresponding winning
ticket could be stretched (or squeezed) into a subnetwork for another deeper
(or shallower) network from the same family, whose performance is nearly the
same competitive as the latter's winning ticket directly found by IMP. We have
also extensively compared E-LTH with pruning-at-initialization and dynamic
sparse training methods, as well as discussed the generalizability of E-LTH to
different model families, layer types, and across datasets. Code is available
at this https URL

本文提出了一种弹性彩票假设（Elastic Lottery Ticket Hypothesis），证明了通过适当地调整一种深度学习神经网络的层级结构，可以从同一类神经网络的另一个较深或较浅网络中拉伸或压缩其获胜彩票，从而实现相当于 IMP 直接发现的性能。

弹性彩票假设

The Elastic Lottery Ticket Hypothesis

Pruning the parameters of deep neural networks has generated intense interest
due to potential savings in time, memory and energy both during training and at
test time. Recent works have identified, through an expensive sequence of
training and pruning cycles, the existence of winning lottery tickets or sparse
trainable subnetworks at initialization. This raises a foundational question:
can we identify highly sparse trainable subnetworks at initialization, without
ever training, or indeed without ever looking at the data? We provide an
affirmative answer to this question through theory driven algorithm design. We
first mathematically formulate and experimentally verify a conservation law
that explains why existing gradient-based pruning algorithms at initialization
suffer from layer-collapse, the premature pruning of an entire layer rendering
a network untrainable. This theory also elucidates how layer-collapse can be
entirely avoided, motivating a novel pruning algorithm Iterative Synaptic Flow
Pruning (SynFlow). This algorithm can be interpreted as preserving the total
flow of synaptic strengths through the network at initialization subject to a
sparsity constraint. Notably, this algorithm makes no reference to the training
data and consistently competes with or outperforms existing state-of-the-art
pruning algorithms at initialization over a range of models (VGG and ResNet),
datasets (CIFAR-10/100 and Tiny ImageNet), and sparsity constraints (up to
99.99 percent). Thus our data-agnostic pruning algorithm challenges the
existing paradigm that, at initialization, data must be used to quantify which
synapses are important.

我们提出了一种不需要训练数据也能够在初始阶段识别高度稀疏的可训练子网络的剪枝算法，该算法基于神经突触流动原理并叫做迭代神经突触流剪枝算法（SynFlow），在多种模型、数据集和稀疏约束条件下，表现出和现有最先进的基于梯度的剪枝算法相当或更好的结果，成功挑战了基于训练数据来定量判断哪些突触重要的传统范式。