Recent advances in the sparse neural network literature have made it possible
to prune many large feed forward and convolutional networks with only a small
quantity of data. Yet, these same techniques often falter when applied to the
problem of recovering sparse recurrent networks. These failures are
quantitative: when pruned with recent techniques, RNNs typically obtain worse
performance than they do under a simple random pruning scheme. The failures are
also qualitative: the distribution of active weights in a pruned LSTM or GRU
network tend to be concentrated in specific neurons and gates, and not well
dispersed across the entire architecture. We seek to rectify both the
quantitative and qualitative issues with recurrent network pruning by
introducing a new recurrent pruning objective derived from the spectrum of the
recurrent Jacobian. Our objective is data efficient (requiring only 64 data
points to prune the network), easy to implement, and produces 95% sparse GRUs
that significantly improve on existing baselines. We evaluate on sequential
MNIST, Billion Words, and Wikitext.

这篇论文通过引入一种基于循环雅可比谱的新的循环网络修剪目标，提出了一种高效数据、易于实现的处理方案，能够解决循环网络修剪中的定量和定性问题，并在连续 MNIST、十亿词、Wikitext 等任务中取得了长足进展。

通过雅可比谱评估一次性剪枝递归神经网络

One-Shot Pruning of Recurrent Neural Networks by Jacobian Spectrum  Evaluation

Recent work has shown that tight concentration of the entire spectrum of
singular values of a deep network's input-output Jacobian around one at
initialization can speed up learning by orders of magnitude. Therefore, to
guide important design choices, it is important to build a full theoretical
understanding of the spectra of Jacobians at initialization. To this end, we
leverage powerful tools from free probability theory to provide a detailed
analytic understanding of how a deep network's Jacobian spectrum depends on
various hyperparameters including the nonlinearity, the weight and bias
distributions, and the depth. For a variety of nonlinearities, our work reveals
the emergence of new universal limiting spectral distributions that remain
concentrated around one even as the depth goes to infinity.

利用自由概率理论，对深度网络输入 - 输出 Jacobian 谱的结构进行研究，探讨非线性、权值、偏置分布、深度等超参数对 Jacobian 谱的影响。我们发现，不同的非线性函数都符合一些新的普适极限谱分布，即使深度接近无穷，这些分布也可以牢固地集中于 1 附近，从而可以为深度网络的设计提供重要的指导。