Modern deep neural networks are typically highly overparameterized. Pruning
techniques are able to remove a significant fraction of network parameters with
little loss in accuracy. Recently, techniques based on dynamic reallocation of
non-zero parameters have emerged, allowing direct training of sparse networks
without having to pre-train a large dense model. Here we present a novel
dynamic sparse reparameterization method that addresses the limitations of
previous techniques such as high computational cost and the need for manual
configuration of the number of free parameters allocated to each layer. We
evaluate the performance of dynamic reallocation methods in training deep
convolutional networks and show that our method outperforms previous static and
dynamic reparameterization methods, yielding the best accuracy for a fixed
parameter budget, on par with accuracies obtained by iteratively pruning a
pre-trained dense model. We further investigated the mechanisms underlying the
superior generalization performance of the resultant sparse networks. We found
that neither the structure, nor the initialization of the non-zero parameters
were sufficient to explain the superior performance. Rather, effective learning
crucially depended on the continuous exploration of the sparse network
structure space during training. Our work suggests that exploring structural
degrees of freedom during training is more effective than adding extra
parameters to the network.

介绍了一种新的动态稀疏重参数化方法，能够更有效地训练深度卷积神经网络，在固定的参数预算下达到最佳准确率，并发现在训练过程中探索结构自由度比增加额外的参数对网络性能的提升更为有效。