Fully exploiting the learning capacity of neural networks requires
overparameterized dense networks. On the other side, directly training sparse
neural networks typically results in unsatisfactory performance. Lottery Ticket
Hypothesis (LTH) provides a novel view to investigate sparse network training
and maintain its capacity. Concretely, it claims there exist winning tickets
from a randomly initialized network found by iterative magnitude pruning and
preserving promising trainability (or we say being in trainable condition). In
this work, we regard the winning ticket from LTH as the subnetwork which is in
trainable condition and its performance as our benchmark, then go from a
complementary direction to articulate the Dual Lottery Ticket Hypothesis
(DLTH): Randomly selected subnetworks from a randomly initialized dense network
can be transformed into a trainable condition and achieve admirable performance
compared with LTH -- random tickets in a given lottery pool can be transformed
into winning tickets. Specifically, by using uniform-randomly selected
subnetworks to represent the general cases, we propose a simple sparse network
training strategy, Random Sparse Network Transformation (RST), to substantiate
our DLTH. Concretely, we introduce a regularization term to borrow learning
capacity and realize information extrusion from the weights which will be
masked. After finishing the transformation for the randomly selected
subnetworks, we conduct the regular finetuning to evaluate the model using fair
comparisons with LTH and other strong baselines. Extensive experiments on
several public datasets and comparisons with competitive approaches validate
our DLTH as well as the effectiveness of the proposed model RST. Our work is
expected to pave a way for inspiring new research directions of sparse network
training in the future. Our code is available at
this https URL

本论文提出 Dual Lottery Ticket Hypothesis 和 Random Sparse Network Transformation 实现稀疏神经网络训练，并通过实验证明了其有效性。