Importance sampling has been successfully used to accelerate stochastic optimization in many convex problems. However, the lack of an efficient way to calculate the importance still hinders its application to Deep Learning. In this paper, we show that the loss value can be used as an alternative importance metric, and propose a way to efficiently approximate it for a deep model, using a small model trained for that purpose in parallel. This method allows in particular to utilize a biased gradient estimate that implicitly optimizes a soft max-loss, and leads to better generalization performance. While such method suffers from a prohibitively high variance of the gradient estimate when using a standard stochastic optimizer, we show that when it is combined with our sampling mechanism, it results in a reliable procedure. We showcase the generality of our method by testing it on both image classification and language modeling tasks using deep convolutional and recurrent neural networks. In particular, in case of CIFAR10 we reach 10% classification error 50 epochs faster than when using uniform sampling.

本文提出一种有效的计算深度学习模型中loss value的方法，它使用小型模型在并行训练时提高了深度学习优化中重要抽样的应用。结果表明，此方法在测试深度卷积和递归神经网络的图像分类和语言建模任务时取得了良好的普适性。

深度神经网络训练的有偏重要性采样