Modern deep neural networks have a large number of parameters, making them
very hard to train. We propose DSD, a dense-sparse-dense training flow, for
regularizing deep neural networks and achieving better optimization
performance. In the first D (Dense) step, we train a dense network to learn
connection weights and importance. In the S (Sparse) step, we regularize the
network by pruning the unimportant connections with small weights and
retraining the network given the sparsity constraint. In the final D (re-Dense)
step, we increase the model capacity by removing the sparsity constraint,
re-initialize the pruned parameters from zero and retrain the whole dense
network. Experiments show that DSD training can improve the performance for a
wide range of CNNs, RNNs and LSTMs on the tasks of image classification,
caption generation and speech recognition. On ImageNet, DSD improved the Top1
accuracy of GoogLeNet by 1.1%, VGG-16 by 4.3%, ResNet-18 by 1.2% and ResNet-50
by 1.1%, respectively. On the WSJ'93 dataset, DSD improved DeepSpeech and
DeepSpeech2 WER by 2.0% and 1.1%. On the Flickr-8K dataset, DSD improved the
NeuralTalk BLEU score by over 1.7. DSD is easy to use in practice: at training
time, DSD incurs only one extra hyper-parameter: the sparsity ratio in the S
step. At testing time, DSD doesn't change the network architecture or incur any
inference overhead. The consistent and significant performance gain of DSD
experiments shows the inadequacy of the current training methods for finding
the best local optimum, while DSD effectively achieves superior optimization
performance for finding a better solution. DSD models are available to download
at this https URL

通过提出 DSD 训练流程，即 Dense-Sparse-Dense（DSD）的训练方法，对深度神经网络进行正则化，以实现更好的优化性能；实验证明，DSD 训练可以提高各种神经网络模型（如 CNN、RNN 和 LSTM）在图像分类、标题生成和语音识别等任务中的性能，并且可以在不改变网络架构或引入任何推理开销的情况下使用。