In recent years, deep neural networks (DNNs) have been applied to various
machine leaning tasks, including image recognition, speech recognition, and
machine translation. However, large DNN models are needed to achieve
state-of-the-art performance, exceeding the capabilities of edge devices. Model
reduction is thus needed for practical use. In this paper, we point out that
deep learning automatically induces group sparsity of weights, in which all
weights connected to an output channel (node) are zero, when training DNNs
under the following three conditions: (1) rectified-linear-unit (ReLU)
activations, (2) an $L_2$-regularized objective function, and (3) the Adam
optimizer. Next, we analyze this behavior both theoretically and
experimentally, and propose a simple model reduction method: eliminate the zero
weights after training the DNN. In experiments on MNIST and CIFAR-10 datasets,
we demonstrate the sparsity with various training setups. Finally, we show that
our method can efficiently reduce the model size and performs well relative to
methods that use a sparsity-inducing regularizer.

本文提出深度神经网络在训练过程中能自动诱导出权重的分组稀疏性，利用此现象我们可以在训练完成之后快速减少模型尺寸，同时不会对模型表现带来太大影响。