We introduce dropout compaction, a novel method for training feed-forward
neural networks which realizes the performance gains of training a large model
with dropout regularization, yet extracts a compact neural network for run-time
efficiency. In the proposed method, we introduce a sparsity-inducing prior on
the per unit dropout retention probability so that the optimizer can
effectively prune hidden units during training. By changing the prior
hyperparameters, we can control the size of the resulting network. We performed
a systematic comparison of dropout compaction and competing methods on several
real-world speech recognition tasks and found that dropout compaction achieved
comparable accuracy with fewer than 50% of the hidden units, translating to a
2.5x speedup in run-time.

介绍了一种称为 Dropout Compression 的新型神经网络训练方法，可以在训练时实现大型模型中的 dropout 正则化性能提升，同时提取出紧凑型的神经网络以实现运行时效率，通过引入对单个神经元 dropout 保留概率的稀疏性先验，可以有效地在训练过程中修剪隐藏单元，并通过调整先验超参数来控制最终网络的大小，并在多个真实的语音识别任务中对丢失压缩和竞争方法进行了系统比较，结果显示，丢失压缩在保持相当准确性的情况下使用不到 50% 的隐藏单元，运行时间可以提速 2.5 倍。