Large, curated, web-crawled corpora play a vital role in training language
models (LMs). They form the lion's share of the training data in virtually all
recent LMs, such as the well-known GPT, LLaMA and XLM-RoBERTa models. However,
despite this importance, relatively little attention has been given to the
quality of these corpora. In this paper, we compare four of the currently most
relevant large, web-crawled corpora (CC100, MaCoCu, mC4 and OSCAR) across
eleven lower-resourced European languages. Our approach is two-fold: first, we
perform an intrinsic evaluation by performing a human evaluation of the quality
of samples taken from different corpora; then, we assess the practical impact
of the qualitative differences by training specific LMs on each of the corpora
and evaluating their performance on downstream tasks. We find that there are
clear differences in quality of the corpora, with MaCoCu and OSCAR obtaining
the best results. However, during the extrinsic evaluation, we actually find
that the CC100 corpus achieves the highest scores. We conclude that, in our
experiments, the quality of the web-crawled corpora does not seem to play a
significant role when training LMs.

经过评估，我们发现大规模网络爬取的语料库对训练语言模型的质量并不起重要作用。