Large-scale corpora play a vital role in the construction of large language
models (LLMs). However, existing LLMs exhibit limited abilities in
understanding low-resource languages, including the minority languages in
China, due to a lack of training data. To improve the accessibility of these
languages, we present MC^2, a Multilingual Corpus of Minority Languages in
China, which is the largest open-source corpus so far. It encompasses four
underrepresented languages, i.e., Tibetan, Uyghur, Kazakh in the Kazakh Arabic
script, and Mongolian in the traditional Mongolian script. Notably, two writing
systems in MC^2 are long neglected in previous corpora. As we identify serious
contamination in the low-resource language split in the existing multilingual
corpora, we propose a quality-centric solution for collecting MC^2,
prioritizing quality and accuracy while enhancing representativeness and
diversity. By in-depth analysis, we demonstrate the new research challenges
MC^2 brings, such as long-text modeling and multiplicity of writing systems. We
hope MC^2 can help enhance the equity of the underrepresented languages in
China and provide a reliable data foundation for further research on
low-resource languages.

MC^2 是迄今为止最大的开源多语言少数民族语言语料库，涵盖了西藏语、维吾尔语、哈萨克语（哈萨克阿拉伯文）和蒙古语（传统蒙古文）等四种弱势语言，着重解决现有多语言语料库中有限的资源语言分割问题，并分析了长文本建模和多重书写系统等带来的新研究挑战。希望 MC^2 能够提高中国弱势语言的平等性，并为低资源语言的进一步研究提供可靠的数据基础。

MC^2: 中国少数民族语言的多语种语料库

MC^2: A Multilingual Corpus of Minority Languages in China

We present GlotScript, an open resource and tool for low resource writing
system identification. GlotScript-R is a resource that provides the attested
writing systems for more than 7,000 languages. It is compiled by aggregating
information from existing writing system resources. GlotScript-T is a writing
system identification tool that covers all 161 Unicode 15.0 scripts. For an
input text, it returns its script distribution where scripts are identified by
ISO 15924 codes. We also present two use cases for GlotScript. First, we
demonstrate that GlotScript supports cleaning multilingual corpora such as mC4
and OSCAR. Second, we analyze the tokenization of a number of language models
such as GPT-4 using GlotScript and provide insights on the coverage of low
resource scripts and languages by each language model. We hope that GlotScript
will become a useful resource for work on low resource languages in the NLP
community. GlotScript-R and GlotScript-T are available at
this https URL

GlotScript 是一个开放资源和工具，用于低资源写作系统识别，提供了超过 7000 种语言的已验证写作系统，并且通过整合现有写作系统资源编制。GlotScript-T 是一个涵盖了 161 个 Unicode 15.0 脚本的写作系统识别工具，用 ISO 15924 代码标识脚本，以输入文本返回其脚本分布。该研究还展示了 GlotScript 的两个用例，支持清理多语言语料库和分析语言模型的标记化，希望成为 NLP 社区中低资源语言研究的有用资源。

GlotScript: 低资源书写系统识别的资源与工具

GlotScript: A Resource and Tool for Low Resource Writing System  Identification

Grapheme-to-phoneme conversion (g2p) is necessary for text-to-speech and
automatic speech recognition systems. Most g2p systems are monolingual: they
require language-specific data or handcrafting of rules. Such systems are
difficult to extend to low resource languages, for which data and handcrafted
rules are not available. As an alternative, we present a neural
sequence-to-sequence approach to g2p which is trained on
spelling--pronunciation pairs in hundreds of languages. The system shares a
single encoder and decoder across all languages, allowing it to utilize the
intrinsic similarities between different writing systems. We show an 11%
improvement in phoneme error rate over an approach based on adapting
high-resource monolingual g2p models to low-resource languages. Our model is
also much more compact relative to previous approaches.

本文提出了使用神经序列到序列模型进行语音转换的方法，这个方法可以用在多种语言上，并且相比于基于高资源单语言模型适应低资源语言的方法，我们的方法在语音识别上的表现有显著提升，同时我们的模型更加紧凑。