Model compression is essential for serving large deep neural nets on devices
with limited resources or applications that require real-time responses. As a
case study, a state-of-the-art neural language model usually consists of one or
more recurrent layers sandwiched between an embedding layer used for
representing input tokens and a softmax layer for generating output tokens. For
problems with a very large vocabulary size, the embedding and the softmax
matrices can account for more than half of the model size. For instance, the
bigLSTM model achieves state-of- the-art performance on the One-Billion-Word
(OBW) dataset with around 800k vocabulary, and its word embedding and softmax
matrices use more than 6GBytes space, and are responsible for over 90% of the
model parameters. In this paper, we propose GroupReduce, a novel compression
method for neural language models, based on vocabulary-partition (block) based
low-rank matrix approximation and the inherent frequency distribution of tokens
(the power-law distribution of words). The experimental results show our method
can significantly outperform traditional compression methods such as low-rank
approximation and pruning. On the OBW dataset, our method achieved 6.6 times
compression rate for the embedding and softmax matrices, and when combined with
quantization, our method can achieve 26 times compression rate, which
translates to a factor of 12.8 times compression for the entire model with very
little degradation in perplexity.

本文提出了一种新的神经语言模型压缩方法，基于基于词汇划分的低秩矩阵逼近和词汇分布的幂律分布，相比传统压缩方法更优，在 OBW 数据集上实现了 6.6 倍的压缩率，采用量化可以实现 26 倍的压缩率，而且模型困惑度几乎没有降低。