Large text corpora are increasingly important for a wide variety of Natural
Language Processing (NLP) tasks, and automatic language identification (LangID)
is a core technology needed to collect such datasets in a multilingual context.
LangID is largely treated as solved in the literature, with models reported
that achieve over 90% average F1 on as many as 1,366 languages. We train LangID
models on up to 1,629 languages with comparable quality on held-out test sets,
but find that human-judged LangID accuracy for web-crawl text corpora created
using these models is only around 5% for many lower-resource languages,
suggesting a need for more robust evaluation. Further analysis revealed a
variety of error modes, arising from domain mismatch, class imbalance, language
similarity, and insufficiently expressive models. We propose two classes of
techniques to mitigate these errors: wordlist-based tunable-precision filters
(for which we release curated lists in about 500 languages) and
transformer-based semi-supervised LangID models, which increase median dataset
precision from 5.5% to 71.2%. These techniques enable us to create an initial
data set covering 100K or more relatively clean sentences in each of 500+
languages, paving the way towards a 1,000-language web text corpus.

本研究重点讨论了大型文本语料库在自然语言处理任务中的重要性以及在多语言语境下收集这些数据集所需的核心技术 —— 自动语言识别，该文介绍了使用基于词表的精度可调过滤器和基于 Transformer 的半监督语言识别模型来提高数据集质量的方法，为接下来创建 1,000 种语言的网络文本语料库铺平了道路。