Weight pruning is an effective technique to reduce the model size and
inference time for deep neural networks in real-world deployments. However,
since magnitudes and relative importance of weights are very different for
different layers of a neural network, existing methods rely on either manual
tuning or handcrafted heuristic rules to find appropriate pruning rates
individually for each layer. This approach generally leads to suboptimal
performance. In this paper, by directly working on the probability space, we
propose an effective network sparsification method called {\it probabilistic
masking} (ProbMask), which solves a natural sparsification formulation under
global sparsity constraint. The key idea is to use probability as a global
criterion for all layers to measure the weight importance. An appealing feature
of ProbMask is that the amounts of weight redundancy can be learned
automatically via our constraint and thus we avoid the problem of tuning
pruning rates individually for different layers in a network. Extensive
experimental results on CIFAR-10/100 and ImageNet demonstrate that our method
is highly effective, and can outperform previous state-of-the-art methods by a
significant margin, especially in the high pruning rate situation. Notably, the
gap of Top-1 accuracy between our ProbMask and existing methods can be up to
10\%. As a by-product, we show ProbMask is also highly effective in identifying
supermasks, which are subnetworks with high performance in a randomly weighted
dense neural network.

该研究提出了一种名为 ProbMask 的基于概率空间的有效网络稀疏化方法，通过使用概率作为全局标准来度量权重重要性。 通过我们的约束，可以自动学习权重冗余度，避免为不同层调整剪枝率的问题，表现优异在 CIFAR-10/100 和 ImageNet 等数据集上，尤其是在高剪枝率情况下超过现有的最先进方法。