We create publicly available language identification (LID) datasets and
models in all 22 Indian languages listed in the Indian constitution in both
native-script and romanized text. First, we create Bhasha-Abhijnaanam, a
language identification test set for native-script as well as romanized text
which spans all 22 Indic languages. We also train IndicLID, a language
identifier for all the above-mentioned languages in both native and romanized
script. For native-script text, it has better language coverage than existing
LIDs and is competitive or better than other LIDs. IndicLID is the first LID
for romanized text in Indian languages. Two major challenges for romanized text
LID are the lack of training data and low-LID performance when languages are
similar. We provide simple and effective solutions to these problems. In
general, there has been limited work on romanized text in any language, and our
findings are relevant to other languages that need romanized language
identification. Our models are publicly available at
this https URL under open-source licenses. Our training
and test sets are also publicly available at
this https URL under open-source
licenses.

我们为所有列在印度宪法中的 22 种印度语言创建了用于母语和罗马化文本的公开语言识别（LID）数据集和模型，并为类似的其他语言提供了解决罗马化文本 LID 中缺乏训练数据和低 LID 性能的简单有效的解决方案。