Pretraining data selection has the potential to improve language model
pretraining efficiency by utilizing higher-quality data from massive web data
corpora. Current data selection methods, which rely on either hand-crafted
rules or larger reference models, are conducted statically and do not capture
the evolving data preferences during pretraining. In this paper, we introduce
model-aware data selection with data influence models (MATES), where a data
influence model continuously adapts to the evolving data preferences of the
pretraining model and then selects the data most effective for the current
pretraining progress. Specifically, we fine-tune a small data influence model
to approximate oracle data preference signals collected by locally probing the
pretraining model and to select data accordingly for the next pretraining
stage. Experiments on Pythia and the C4 dataset demonstrate that MATES
significantly outperforms random data selection on extensive downstream tasks
in both zero- and few-shot settings. It doubles the gains achieved by recent
data selection approaches that leverage larger reference models and reduces the
total FLOPs required to reach certain performances by half. Further analysis
validates the ever-changing data preferences of pretraining models and the
effectiveness of our data influence models to capture them. Our code is
open-sourced at this https URL

使用模型感知数据选择方法 (MATES) 能够在预训练模型中持续调整数据选择，契合当前预训练进展并在后续任务中显著提高性能。