We introduce Aksharantar, the largest publicly available transliteration
dataset for 21 Indic languages containing 26 million transliteration pairs. We
build this dataset by mining transliteration pairs from large monolingual and
parallel corpora, as well as collecting transliterations from human annotators
to ensure diversity of words and representation of low-resource languages. We
introduce a new, large, diverse testset for Indic language transliteration
containing 103k words pairs spanning 19 languages that enables fine-grained
analysis of transliteration models.
We train the IndicXlit model on the Aksharantar training set. IndicXlit is a
single transformer-based multilingual transliteration model for roman to Indic
script conversion supporting 21 Indic languages. It achieves state-of-the art
results on the Dakshina testset, and establishes strong baselines on the
Aksharantar testset released along with this work.
All the datasets and models are publicly available at
this https URL. We hope the availability of these
large-scale, open resources will spur innovation for Indic language
transliteration and downstream applications.

本文介绍了 Aksharantar，它是 21 种印度语言中最大的公开可用的音译数据集，包含 2600 万个音译对。我们使用从大型单语和平行语料库中挖掘音译对，并采集人工注释以确保具有不同单词和低资源语言的多样性。我们在 Aksharantar 训练集上训练了 IndicXlit 模型，它是一种单一的基于 transformer 的多语言音译模型，支持 21 种印度语言。它在 Dakshina 测试集上取得了最先进的结果，并在与此工作一起发布的 Aksharantar 测试集上建立了强大的基线。