The ever-growing diversity of pre-training text corpora has equipped language
models with generalization capabilities across various downstream tasks.
However, such diverse datasets are often too large for academic budgets; hence,
most research on Transformer architectures, training procedures, optimizers,
etc. gets conducted on smaller, homogeneous datasets. To this end, we present
The MiniPile Challenge, where one pre-trains a language model on a diverse text
corpus containing at most 1M documents. MiniPile is a 6GB subset of the
deduplicated 825GB The Pile corpus. To curate MiniPile, we perform a simple,
three-step data filtering process: we (1) infer embeddings for all documents of
the Pile, (2) cluster the embedding space using $k$-means, and (3) filter out
low-quality clusters. To verify MiniPile's suitability for language model
pre-training, we use it to pre-train a BERT and T5 model, yielding a
performance drop of only $1.9\%$/$2.5\%$ on the GLUE and SNI benchmarks
compared to the original pre-trained checkpoints trained on $2.6$x/$745$x the
amount of data. MiniPile is available at
this https URL

本文提出 MiniPile 挑战，呈现一种使用文本语料库的小数据集进行语言模型预训练的方法，其适用性通过在 GLUE 和 SNI 基准测试中得到论证。