Recurrent neural networks (RNNs) have recently achieved remarkable successes
in a number of applications. However, the huge sizes and computational burden
of these models make it difficult for their deployment on edge devices. A
practically effective approach is to reduce the overall storage and computation
costs of RNNs by network pruning techniques. Despite their successful
applications, those pruning methods based on Lasso either produce irregular
sparse patterns in weight matrices, which is not helpful in practical speedup.
To address these issues, we propose structured pruning method through neuron
selection which can reduce the sizes of basic structures of RNNs. More
specifically, we introduce two sets of binary random variables, which can be
interpreted as gates or switches to the input neurons and the hidden neurons,
respectively. We demonstrate that the corresponding optimization problem can be
addressed by minimizing the L0 norm of the weight matrix. Finally, experimental
results on language modeling and machine reading comprehension tasks have
indicated the advantages of the proposed method in comparison with
state-of-the-art pruning competitors. In particular, nearly 20 x practical
speedup during inference was achieved without losing performance for language
model on the Penn TreeBank dataset, indicating the promising performance of the
proposed method

通过神经元选择的结构化剪枝方法，可以通过减小基本结构的尺寸来减少 RNN 的存储和计算成本，并利用 L0 范数优化，取得了语言建模等任务上的优异表现。