Model pruning seeks to induce sparsity in a deep neural network's various
connection matrices, thereby reducing the number of nonzero-valued parameters
in the model. Recent reports (Han et al., 2015; Narang et al., 2017) prune deep
networks at the cost of only a marginal loss in accuracy and achieve a sizable
reduction in model size. This hints at the possibility that the baseline models
in these experiments are perhaps severely over-parameterized at the outset and
a viable alternative for model compression might be to simply reduce the number
of hidden units while maintaining the model's dense connection structure,
exposing a similar trade-off in model size and accuracy. We investigate these
two distinct paths for model compression within the context of energy-efficient
inference in resource-constrained environments and propose a new gradual
pruning technique that is simple and straightforward to apply across a variety
of models/datasets with minimal tuning and can be seamlessly incorporated
within the training process. We compare the accuracy of large, but pruned
models (large-sparse) and their smaller, but dense (small-dense) counterparts
with identical memory footprint. Across a broad range of neural network
architectures (deep CNNs, stacked LSTM, and seq2seq LSTM models), we find
large-sparse models to consistently outperform small-dense models and achieve
up to 10x reduction in number of non-zero parameters with minimal loss in
accuracy.

本文探讨在资源受限环境下，通过模型剪枝来压缩神经网络模型的方法，提出了一种简单、直接、易于应用的逐渐剪枝技术，并在多个模型 / 数据集上进行了比较，发现大型稀疏模型在保持较高精度的同时可减少 10 倍的参数数量。