Large Language Model (LLM) pre-training exhausts an ever growing compute
budget, yet recent research has demonstrated that careful document selection
enables comparable model quality with only a fraction of the FLOPs. Inspired by
efforts suggesting that domain-specific training document selection is in fact
an interpretable process [Gunasekar et al., 2023], as well as research showing
that instruction-finetuned LLMs are adept zero-shot data labelers [Gilardi et
al.,2023], we explore a promising direction for scalable general-domain
document selection; employing a prompted LLM as a document grader, we distill
quality labels into a classifier model, which is applied at scale to a large,
and already heavily-filtered, web-crawl-derived corpus autonomously. Following
the guidance of this classifier, we drop 75% of the corpus and train LLMs on
the remaining data. Results across multiple benchmarks show that: 1. Filtering
allows us to quality-match a model trained on the full corpus across diverse
benchmarks with at most 70% of the FLOPs, 2. More capable LLM labelers and
classifier models lead to better results that are less sensitive to the
labeler's prompt, 3. In-context learning helps to boost the performance of
less-capable labeling models. In all cases we use open-source datasets, models,
recipes, and evaluation frameworks, so that results can be reproduced by the
community.

LLM 预训练模型通过精心选择文档，可以在仅使用部分 FLOPs 的情况下实现与完整训练相当的模型质量；通过使用提示的 LLM 作为文档评分器，我们将质量标签提取并应用于大规模的互联网抓取数据集，以自动筛除部分文档，从而更好地匹配性能，并通过在上下文中学习来提高标签模型的性能。

大规模语言模型引导的文档选择

Large Language Model-guided Document Selection

We present a novel method for obtaining high-quality, domain-targeted
multiple choice questions from crowd workers. Generating these questions can be
difficult without trading away originality, relevance or diversity in the
answer options. Our method addresses these problems by leveraging a large
corpus of domain-specific text and a small set of existing questions. It
produces model suggestions for document selection and answer distractor choice
which aid the human question generation process. With this method we have
assembled SciQ, a dataset of 13.7K multiple choice science exam questions
(Dataset available at this http URL). We demonstrate that the
method produces in-domain questions by providing an analysis of this new
dataset and by showing that humans cannot distinguish the crowdsourced
questions from original questions. When using SciQ as additional training data
to existing questions, we observe accuracy improvements on real science exams.

本文提出一种新的方法来从众包工作者那里获得高质量、针对领域的多项选择题，该方法通过利用大量的领域特定文本和一小部分现有问题，产生文档选择和答案干扰选项的模型建议，辅助人类提问生成过程。使用这种方法，我们已经组装了 13.7K 的多项选择科学考试题的 SciQ 数据集，并通过提供对这个新数据集的分析和展示人们无法区分众包问题与原问题，证明了这种方法可以产生领域内的问题。当将 SciQ 用作现有问题的附加训练数据时，在真实科学考试中观察到了准确度的提高。