The mixture proportions of pretraining data domains (e.g., Wikipedia, books,
web text) greatly affect language model (LM) performance. In this paper, we
propose Domain Reweighting with Minimax Optimization (DoReMi), which first
trains a small proxy model using group distributionally robust optimization
(Group DRO) over domains to produce domain weights (mixture proportions)
without knowledge of downstream tasks. We then resample a dataset with these
domain weights and train a larger, full-sized model. In our experiments, we use
DoReMi on a 280M-parameter proxy model to find domain weights for training an
8B-parameter model (30x larger) more efficiently. On The Pile, DoReMi improves
perplexity across all domains, even when it downweights a domain. DoReMi
improves average few-shot downstream accuracy by 6.5% over a baseline model
trained using The Pile's default domain weights and reaches the baseline
accuracy with 2.6x fewer training steps. On the GLaM dataset, DoReMi, which has
no knowledge of downstream tasks, even matches the performance of using domain
weights tuned on downstream tasks.

本篇论文介绍了一种名为 DoReMi 的方法，用于通过预训练数据域的混合比例来提高语言模型性能，该方法先使用集合分布稳健优化（Group DRO）对数据域进行小型代理模型训练来生成域权重，然后使用这些域权重对数据集进行再采样并训练更大的模型。在实验中，该方法提高了在所有域上的 perplexity ，并在几乎不增加训练步骤的情况下将平均 downstream accuracy 提高了 6.5％。