Recurrent neural networks (RNNs), including long short-term memory (LSTM)
RNNs, have produced state-of-the-art results on a variety of speech recognition
tasks. However, these models are often too large in size for deployment on
mobile devices with memory and latency constraints. In this work, we study
mechanisms for learning compact RNNs and LSTMs via low-rank factorizations and
parameter sharing schemes. Our goal is to investigate redundancies in recurrent
architectures where compression can be admitted without losing performance. A
hybrid strategy of using structured matrices in the bottom layers and shared
low-rank factors on the top layers is found to be particularly effective,
reducing the parameters of a standard LSTM by 75%, at a small cost of 0.3%
increase in WER, on a 2,000-hr English Voice Search task.

本文介绍了一种利用低秩分解和参数共享技术来学习紧凑的循环神经网络（LSTM）的机制，探究了在不损失性能的情况下完善紧凑结构的可能性，并且发现在底层使用结构矩阵，在顶层使用共享低秩因子的混合策略特别有效，在 2000 小时英语语音搜索任务上将标准 LSTM 的参数减少了 75%，仅仅增加了 0.3% 的误差率。