We show implicit filter level sparsity manifests in convolutional neural
networks (CNNs) which employ Batch Normalization and ReLU activation, and are
trained with adaptive gradient descent techniques and L2 regularization or
weight decay. Through an extensive empirical study (Mehta et al., 2019) we
hypothesize the mechanism behind the sparsification process, and find
surprising links to certain filter sparsification heuristics proposed in
literature. Emergence of, and the subsequent pruning of selective features is
observed to be one of the contributing mechanisms, leading to feature sparsity
at par or better than certain explicit sparsification / pruning approaches. In
this workshop article we summarize our findings, and point out corollaries of
selective-featurepenalization which could also be employed as heuristics for
filter pruning

研究发现，在采用批归一化和 ReLU 激活以及使用自适应梯度下降技术和 L2 正则化或权值衰减训练的卷积神经网络中，存在着隐式过滤器级别稀疏性，这可能与某些在文献中提出的过滤器稀疏化启发式的特定联系有关。进一步观察表明，特定功能的出现和随后的修剪是导致特征稀疏化的机制之一，导致的特征稀疏性与某些显式稀疏化 / 修剪方法相当甚至更好。本文总结了研究发现，并指出了选择性功能处罚的推论，这也可作为过滤器修剪的启发式方法。

卷积神经网络中的隐式滤波器稀疏化

Implicit Filter Sparsification In Convolutional Neural Networks

We investigate filter level sparsity that emerges in convolutional neural
networks (CNNs) which employ Batch Normalization and ReLU activation, and are
trained with adaptive gradient descent techniques and L2 regularization or
weight decay. We conduct an extensive experimental study casting our initial
findings into hypotheses and conclusions about the mechanisms underlying the
emergent filter level sparsity. This study allows new insight into the
performance gap obeserved between adapative and non-adaptive gradient descent
methods in practice. Further, analysis of the effect of training strategies and
hyperparameters on the sparsity leads to practical suggestions in designing CNN
training strategies enabling us to explore the tradeoffs between feature
selectivity, network capacity, and generalization performance. Lastly, we show
that the implicit sparsity can be harnessed for neural network speedup at par
or better than explicit sparsification / pruning approaches, with no
modifications to the typical training pipeline required.

研究表明，使用 Batch Normalization 和 ReLU 激活的卷积神经网络，在采用适应性梯度下降和 L2 正则化或权重衰减训练的情况下，会出现滤波器级别的稀疏性，这种隐式的稀疏性可以利用以达到神经网络加速的效果。