Recent work has demonstrated that increased training dataset diversity
improves general cross-domain knowledge and downstream generalization
capability for large-scale language models. With this in mind, we present
\textit{the Pile}: an 825 GiB English text corpus targeted at training
large-scale language models. The Pile is constructed from 22 diverse
high-quality subsets -- both existing and newly constructed -- many of which
derive from academic or professional sources. Our evaluation of the untuned
performance of GPT-2 and GPT-3 on the Pile shows that these models struggle on
many of its components, such as academic writing. Conversely, models trained on
the Pile improve significantly over both Raw CC and CC-100 on all components of
the Pile, while improving performance on downstream evaluations. Through an
in-depth exploratory analysis, we document potentially concerning aspects of
the data for prospective users. We make publicly available the code used in its
construction.

本研究使用 825GB 的英文文本语料库，旨在训练大规模语言模型，通过 GPT-2 和 GPT-3 的预实验发现，这些模型在处理学术写作等特定组件上表现不佳，而在训练了语料库后在所有组件上都有了显著提高。同时，我们对数据进行了深入分析，提供代码用于其构建。