Several recent papers have published good solutions for language
identification (LID) for about 300 high-resource and medium-resource languages.
However, there is no LID available that (i) covers a wide range of low-resource
languages, (ii) is rigorously evaluated and reliable and (iii) efficient and
easy to use. Here, we publish GlotLID-M, an LID model that satisfies the
desiderata of wide coverage, reliability and efficiency. It identifies 1665
languages, a large increase in coverage compared to prior work. In our
experiments, GlotLID-M outperforms four baselines (CLD3, FT176, OpenLID and
NLLB) when balancing F1 and false positive rate (FPR). We analyze the unique
challenges that low-resource LID poses: incorrect corpus metadata, leakage from
high-resource languages, difficulty separating closely related languages,
handling of macrolanguage vs varieties and in general noisy data. We hope that
integrating GlotLID-M into dataset creation pipelines will improve quality and
enhance accessibility of NLP technology for low-resource languages and
cultures. GlotLID-M model, code, and list of data sources are available:
this https URL

GlotLID-M 是一种具备广泛覆盖、可靠高效、用于辨识低资源语言的 LID 模型，通过分析语料元数据问题、高资源语言泄漏、近似语言辨别困难、处理大语种与方言等方面的挑战，希望将其整合到数据集创建过程中，提高低资源语言与文化的 NLP 技术的质量和可用性。