Neural Machine Translation models are extremely data and compute-hungry.
However, not all data points contribute equally to model training and
generalization. Data pruning to remove the low-value data points has the
benefit of drastically reducing the compute budget without significant drop in
model performance. In this paper, we propose a new data pruning technique:
Checkpoints Across Time (CAT), that leverages early model training dynamics to
identify the most relevant data points for model performance. We benchmark CAT
against several data pruning techniques including COMET-QE, LASER and LaBSE. We
find that CAT outperforms the benchmarks on Indo-European languages on multiple
test sets. When applied to English-German, English-French and English-Swahili
translation tasks, CAT achieves comparable performance to using the full
dataset, while pruning up to 50% of training data. We inspect the data points
that CAT selects and find that it tends to favour longer sentences and
sentences with unique or rare words.

我们提出了一种新的数据修剪技术：Checkpoints Across Time (CAT)，通过利用早期模型训练动态来识别对模型性能最相关的数据点，实现了在减少 50% 的训练数据的同时，与使用完整数据集的性能相当，且优于其他数据修剪技术。