It is well known that selecting samples with large losses/gradients can
significantly reduce the number of training steps. However, the selection
overhead is often too high to yield any meaningful gains in terms of overall
training time. In this work, we focus on the greedy approach of selecting
samples with large \textit{approximate losses} instead of exact losses in order
to reduce the selection overhead. For smooth convex losses, we show that such a
greedy strategy can converge to a constant factor of the minimum value of the
average loss in fewer iterations than the standard approach of random
selection. We also theoretically quantify the effect of the approximation
level. We then develop SIFT which uses early exiting to obtain approximate
losses with an intermediate layer's representations for sample selection. We
evaluate SIFT on the task of training a 110M parameter 12-layer BERT base model
and show significant gains (in terms of training hours and number of
backpropagation steps) without any optimized implementation over vanilla
training. For e.g., to reach 64% validation accuracy, SIFT with exit at the
first layer takes ~43 hours compared to ~57 hours of vanilla training.

通过选择具有大梯度 / 损失的样本，可以显著减少训练步骤，本文提出了一种基于贪婪法选择具有大近似损失的样本来降低选择开销，并在训练 BERT 模型上展示了显著的训练时间节约和收敛速度提升。