This paper introduces Selective-Backprop, a technique that accelerates the
training of deep neural networks (DNNs) by prioritizing examples with high loss
at each iteration. Selective-Backprop uses the output of a training example's
forward pass to decide whether to use that example to compute gradients and
update parameters, or to skip immediately to the next example. By reducing the
number of computationally-expensive backpropagation steps performed,
Selective-Backprop accelerates training. Evaluation on CIFAR10, CIFAR100, and
SVHN, across a variety of modern image models, shows that Selective-Backprop
converges to target error rates up to 3.5x faster than with standard SGD and
between 1.02--1.8x faster than a state-of-the-art importance sampling approach.
Further acceleration of 26% can be achieved by using stale forward pass results
for selection, thus also skipping forward passes of low priority examples.

介绍了 Selective-Backprop 技术，该技术可以通过在每次迭代中优先处理损失较大的示例来加速深度神经网络训练，通过在多个现代图像模型上的评估表明，相比标准 SGD 和最先进的重要性采样方法，Selective-Backprop 可以以快至多达 3.5 倍的速度收敛到目标误差率。