Building pretrained language models is considered expensive and
data-intensive, but must we increase dataset size to achieve better
performance? We propose an alternative to larger training sets by automatically
identifying smaller yet domain-representative subsets. We extend Cynical Data
Selection, a statistical sentence scoring method that conditions on a
representative target domain corpus. As an example, we treat the OntoNotes
corpus as a target domain and pretrain a RoBERTa-like encoder from a cynically
selected subset of the Pile. On both perplexity and across several downstream
tasks in the target domain, it consistently outperforms random selection with
20x less data, 3x fewer training iterations, and 2x less estimated cloud
compute cost, validating the recipe of automatic document selection for LM
pretraining.

通过自动识别小但代表领域的子集，我们提出了一种替代更大的训练集的方法来预训练语言模型，拓展了一种基于统计的句子评分方法，以代表性目标领域语料库作为条件，例如，我们将 OntoNotes 语料库视为目标域并从 Pile cynically 选出一个子集对 RoBERTa 类编码器进行预训练，在困惑度和目标域中的多个下游任务上，它始终优于随机选择，但数据量少 20 倍，训练迭代次数少 3 倍，估计的云计算成本少 2 倍，验证了自动文档选择的预训练语言模型食谱。