The vast majority of non-English corpora are derived from automatically
filtered versions of CommonCrawl. While prior work has identified major issues
on the quality of these datasets (Kreutzer et al., 2021), it is not clear how
this impacts downstream performance. Taking representation learning in Basque
as a case study, we explore tailored crawling (manually identifying and
scraping websites with high-quality content) as an alternative to filtering
CommonCrawl. Our new corpus, called EusCrawl, is similar in size to the Basque
portion of popular multilingual corpora like CC100 and mC4, yet it has a much
higher quality according to native annotators. For instance, 66% of documents
are rated as high-quality for EusCrawl, in contrast with <33% for both mC4 and
CC100. Nevertheless, we obtain similar results on downstream NLU tasks
regardless of the corpus used for pre-training. Our work suggests that NLU
performance in low-resource languages is not primarily constrained by the
quality of the data, and other factors like corpus size and domain coverage can
play a more important role.

通过对巴斯克语的表征学习进行案例研究，我们探索了精细爬取作为替代 CommonCrawl 的方法。尽管我们的新语料库质量更高，而且在巴斯克语广泛的语料库中取得了类似的结果，但我们的工作表明，低资源语言的 NLU 性能不是主要受到数据质量的限制，而是与语料库的大小和领域覆盖等其他因素有更重要的关系。