As there is a scarcity of large representative corpora for most languages, it
is important for Multilingual Language Models (MLLM) to extract the most out of
existing corpora. In this regard, script diversity presents a challenge to
MLLMs by reducing lexical overlap among closely related languages. Therefore,
transliterating closely related languages that use different writing scripts to
a common script may improve the downstream task performance of MLLMs. In this
paper, we pretrain two ALBERT models to empirically measure the effect of
transliteration on MLLMs. We specifically focus on the Indo-Aryan language
family, which has the highest script diversity in the world. Afterward, we
evaluate our models on the IndicGLUE benchmark. We perform Mann-Whitney U test
to rigorously verify whether the effect of transliteration is significant or
not. We find that transliteration benefits the low-resource languages without
negatively affecting the comparatively high-resource languages. We also measure
the cross-lingual representation similarity (CLRS) of the models using centered
kernel alignment (CKA) on parallel sentences of eight languages from the
FLORES-101 dataset. We find that the hidden representations of the
transliteration-based model have higher and more stable CLRS scores. Our code
is available at Github (this http URL) and Hugging Face
Hub (this http URL and
this http URL).

本文研究了在多语言语言模型中应用转写对性能的影响，并在 IndicGLUE 基准测试集上评估了两个 ALBERT 模型，结果显示转写使得低资源语言的性能得到了提升，且基于转写的模型具有更高和更稳定的交叉语言表示相似度分数。

音译是否有助于多语言语言建模？

Does Transliteration Help Multilingual Language Modeling?

We explore the impact of leveraging the relatedness of languages that belong
to the same family in NLP models using multilingual fine-tuning. We hypothesize
and validate that multilingual fine-tuning of pre-trained language models can
yield better performance on downstream NLP applications, compared to models
fine-tuned on individual languages. A first of its kind detailed study is
presented to track performance change as languages are added to a base language
in a graded and greedy (in the sense of best boost of performance) manner;
which reveals that careful selection of subset of related languages can
significantly improve performance than utilizing all related languages. The
Indo-Aryan (IA) language family is chosen for the study, the exact languages
being Bengali, Gujarati, Hindi, Marathi, Oriya, Punjabi and Urdu. The script
barrier is crossed by simple rule-based transliteration of the text of all
languages to Devanagari. Experiments are performed on mBERT, IndicBERT, MuRIL
and two RoBERTa-based LMs, the last two being pre-trained by us. Low resource
languages, such as Oriya and Punjabi, are found to be the largest beneficiaries
of multilingual fine-tuning. Textual Entailment, Entity Classification, Section
Title Prediction, tasks of IndicGLUE and POS tagging form our test bed.
Compared to monolingual fine tuning we get relative performance improvement of
up to 150% in the downstream tasks. The surprise take-away is that for any
language there is a particular combination of other languages which yields the
best performance, and any additional language is in fact detrimental.

本文探讨利用多语言微调技术在 NLP 模型中利用同一语系的相关性对性能的影响，结果表明在微调预训练的多语言模型时，与单独微调每一语言相比，多语言微调可以在下游 NLP 应用程序中获得更好的性能。 印欧 - 雅利安语系（IA）被选为研究对象，实验结果表明精细选择一组相关语言的性能比使用所有相关语言显着提高。