In this paper, we present SwiftLearn, a data-efficient approach to accelerate
training of deep learning models using a subset of data samples selected during
the warm-up stages of training. This subset is selected based on an importance
criteria measured over the entire dataset during warm-up stages, aiming to
preserve the model performance with fewer examples during the rest of training.
The importance measure we propose could be updated during training every once
in a while, to make sure that all of the data samples have a chance to return
to the training loop if they show a higher importance. The model architecture
is unchanged but since the number of data samples controls the number of
forward and backward passes during training, we can reduce the training time by
reducing the number of training samples used in each epoch of training.
Experimental results on a variety of CV and NLP models during both pretraining
and finetuning show that the model performance could be preserved while
achieving a significant speed-up during training. More specifically, BERT
finetuning on GLUE benchmark shows that almost 90% of the data can be dropped
achieving an end-to-end average speedup of 3.36x while keeping the average
accuracy drop less than 0.92%.

SwiftLearn 是一种数据高效的方法，通过在训练的暖启动阶段选择一部分数据样本加速深度学习模型的训练，实现减少训练时间。实验结果表明，在各种计算机视觉和自然语言处理模型的预训练和微调过程中，通过减少训练样本数量可以保持模型性能，并实现显著的加速效果，如 BERT 在 GLUE 基准上可以降低近 90% 的数据，从而实现平均加速比为 3.36 倍，且平均准确度下降小于 0.92%。

SwiftLearn: 使用重要抽样的深度学习模型高效训练方法

SwiftLearn: A Data-Efficient Training Method of Deep Learning Models  using Importance Sampling

Supervised deep learning depends on massive accurately annotated examples,
which is usually impractical in many real-world scenarios. A typical
alternative is learning from multiple noisy annotators. Numerous earlier works
assume that all labels are noisy, while it is usually the case that a few
trusted samples with clean labels are available. This raises the following
important question: how can we effectively use a small amount of trusted data
to facilitate robust classifier learning from multiple annotators? This paper
proposes a data-efficient approach, called \emph{Trustable Co-label Learning}
(TCL), to learn deep classifiers from multiple noisy annotators when a small
set of trusted data is available. This approach follows the coupled-view
learning manner, which jointly learns the data classifier and the label
aggregator. It effectively uses trusted data as a guide to generate trustable
soft labels (termed co-labels). A co-label learning can then be performed by
alternately reannotating the pseudo labels and refining the classifiers. In
addition, we further improve TCL for a special complete data case, where each
instance is labeled by all annotators and the label aggregator is represented
by multilayer neural networks to enhance model capacity. Extensive experiments
on synthetic and real datasets clearly demonstrate the effectiveness and
robustness of the proposed approach. Source code is available at
this https URL

该研究提出了一种数据高效的方法，称为 “可信协同学习”（TCL），在多个嘈杂标注器的情况下从多个受信任的数据中学习深度分类器。这种方法通过联合学习数据分类器和标签聚合器来有效地使用受信任的数据生成可信软标签（称为共标签）。