The vast majority of non-English corpora are derived from automatically
filtered versions of CommonCrawl. While prior work has identified major issues
on the quality of these datasets (Kreutzer et al., 2021), it is not clear how
this impacts downstream performance. Taking representation learning in Basque
as a case study, we explore tailored crawling (manually identifying and
scraping websites with high-quality content) as an alternative to filtering
CommonCrawl. Our new corpus, called EusCrawl, is similar in size to the Basque
portion of popular multilingual corpora like CC100 and mC4, yet it has a much
higher quality according to native annotators. For instance, 66% of documents
are rated as high-quality for EusCrawl, in contrast with <33% for both mC4 and
CC100. Nevertheless, we obtain similar results on downstream NLU tasks
regardless of the corpus used for pre-training. Our work suggests that NLU
performance in low-resource languages is not primarily constrained by the
quality of the data, and other factors like corpus size and domain coverage can
play a more important role.

通过对巴斯克语的表征学习进行案例研究，我们探索了精细爬取作为替代 CommonCrawl 的方法。尽管我们的新语料库质量更高，而且在巴斯克语广泛的语料库中取得了类似的结果，但我们的工作表明，低资源语言的 NLU 性能不是主要受到数据质量的限制，而是与语料库的大小和领域覆盖等其他因素有更重要的关系。

低资源语言的语料库品质真的很重要吗？

Does Corpus Quality Really Matter for Low-Resource Languages?

While BERT is an effective method for learning monolingual sentence
embeddings for semantic similarity and embedding based transfer learning
(Reimers and Gurevych, 2019), BERT based cross-lingual sentence embeddings have
yet to be explored. We systematically investigate methods for learning
multilingual sentence embeddings by combining the best methods for learning
monolingual and cross-lingual representations including: masked language
modeling (MLM), translation language modeling (TLM) (Conneau and Lample, 2019),
dual encoder translation ranking (Guo et al., 2018), and additive margin
softmax (Yang et al., 2019a). We show that introducing a pre-trained
multilingual language model dramatically reduces the amount of parallel
training data required to achieve good performance by 80%. Composing the best
of these methods produces a model that achieves 83.7% bi-text retrieval
accuracy over 112 languages on Tatoeba, well above the 65.5% achieved by
Artetxe and Schwenk (2019b), while still performing competitively on
monolingual transfer learning benchmarks (Conneau and Kiela, 2018). Parallel
data mined from CommonCrawl using our best model is shown to train competitive
NMT models for en-zh and en-de. We publicly release our best multilingual
sentence embedding model for 109+ languages at this https URL

本研究旨在通过探究多种单语和跨语言表示学习方法，如掩码语言建模，翻译语言建模和双编码器翻译排名等，结合预训练的多语言模型来学习多语言句子嵌入，并成功将其用于多语言文本检索和机器翻译任务中。