The training of sparse neural networks is becoming an increasingly important
tool for reducing the computational footprint of models at training and
evaluation, as well enabling the effective scaling up of models. Whereas much
work over the years has been dedicated to specialised pruning techniques,
little attention has been paid to the inherent effect of gradient based
training on model sparsity. In this work, we introduce Powerpropagation, a new
weight-parameterisation for neural networks that leads to inherently sparse
models. Exploiting the behaviour of gradient descent, our method gives rise to
weight updates exhibiting a "rich get richer" dynamic, leaving low-magnitude
parameters largely unaffected by learning. Models trained in this manner
exhibit similar performance, but have a distribution with markedly higher
density at zero, allowing more parameters to be pruned safely. Powerpropagation
is general, intuitive, cheap and straight-forward to implement and can readily
be combined with various other techniques. To highlight its versatility, we
explore it in two very different settings: Firstly, following a recent line of
work, we investigate its effect on sparse training for resource-constrained
settings. Here, we combine Powerpropagation with a traditional weight-pruning
technique as well as recent state-of-the-art sparse-to-sparse algorithms,
showing superior performance on the ImageNet benchmark. Secondly, we advocate
the use of sparsity in overcoming catastrophic forgetting, where compressed
representations allow accommodating a large number of tasks at fixed model
capacity. In all cases our reparameterisation considerably increases the
efficacy of the off-the-shelf methods.

本文介绍了 Powerpropagation 方法，它是一种针对神经网络权重参数化的方法，使用梯度下降时可使权重更新呈现 “富者越富” 的动态，从而产生稀疏模型，在模型性能相似的情况下，分布在零点处的密度提高，使得更多参数可被安全地剪枝，并在两种不同的环境下都表现出优越性。