The data used to pretrain large language models has a decisive impact on a
model's downstream performance, which has led to a large body of work on data
selection methods that aim to automatically determine the most suitable data to
use for pretraining. Existing data selection methods suffer from slow and
computationally expensive processes, a problem amplified by the increasing size
of models and of pretraining datasets. Data mixing, on the other hand, reduces
the complexity of data selection by grouping data points together and
determining sampling probabilities across entire groups. However, data mixing
proportions are typically fixed before training and therefore cannot adapt to
changing training dynamics. To address these limitations, we develop an
efficient algorithm for Online Data Mixing (ODM) that combines elements from
both data selection and data mixing. Based on multi-armed bandit algorithms,
our online approach optimizes the data mixing proportions during training.
Remarkably, our method trains a model that reaches the final perplexity of the
next best method with 19\% fewer training iterations, and improves performance
on the 5-shot MMLU benchmark by 1.9% relative accuracy, while adding negligible
wall-clock time during pretraining.

研究论文通过开发一种高效的在线数据混合算法（ODM），结合了数据选择和数据混合的元素，优化了训练过程中的数据混合比例，提高了大型语言模型的性能。