We investigate filter level sparsity that emerges in convolutional neural
networks (CNNs) which employ Batch Normalization and ReLU activation, and are
trained with adaptive gradient descent techniques and L2 regularization or
weight decay. We conduct an extensive experimental study casting our initial
findings into hypotheses and conclusions about the mechanisms underlying the
emergent filter level sparsity. This study allows new insight into the
performance gap obeserved between adapative and non-adaptive gradient descent
methods in practice. Further, analysis of the effect of training strategies and
hyperparameters on the sparsity leads to practical suggestions in designing CNN
training strategies enabling us to explore the tradeoffs between feature
selectivity, network capacity, and generalization performance. Lastly, we show
that the implicit sparsity can be harnessed for neural network speedup at par
or better than explicit sparsification / pruning approaches, with no
modifications to the typical training pipeline required.

研究表明，使用 Batch Normalization 和 ReLU 激活的卷积神经网络，在采用适应性梯度下降和 L2 正则化或权重衰减训练的情况下，会出现滤波器级别的稀疏性，这种隐式的稀疏性可以利用以达到神经网络加速的效果。