In this paper, we present SwiftLearn, a data-efficient approach to accelerate training of deep learning models using a subset of data samples selected during the warm-up stages of training. This subset is selected based on an importance criteria measured over the entire dataset during warm-up stages, aiming to preserve the model performance with fewer examples during the rest of training. The importance measure we propose could be updated during training every once in a while, to make sure that all of the data samples have a chance to return to the training loop if they show a higher importance. The model architecture is unchanged but since the number of data samples controls the number of forward and backward passes during training, we can reduce the training time by reducing the number of training samples used in each epoch of training. Experimental results on a variety of CV and NLP models during both pretraining and finetuning show that the model performance could be preserved while achieving a significant speed-up during training. More specifically, BERT finetuning on GLUE benchmark shows that almost 90% of the data can be dropped achieving an end-to-end average speedup of 3.36x while keeping the average accuracy drop less than 0.92%.

SwiftLearn是一种数据高效的方法，通过在训练的暖启动阶段选择一部分数据样本加速深度学习模型的训练，实现减少训练时间。实验结果表明，在各种计算机视觉和自然语言处理模型的预训练和微调过程中，通过减少训练样本数量可以保持模型性能，并实现显著的加速效果，如BERT在GLUE基准上可以降低近90%的数据，从而实现平均加速比为3.36倍，且平均准确度下降小于0.92%。

SwiftLearn: 使用重要抽样的深度学习模型高效训练方法