There are 293 scripts representing over 7,000 languages in the written form.
Due to various reasons, many closely related languages use different scripts,
which poses difficulty for multilingual pretrained language models (mPLMs) in
learning crosslingual knowledge through lexical overlap. As a result, mPLMs
present a script barrier: representations from different scripts are located in
different subspaces, which is a strong indicator of why crosslingual transfer
involving languages of different scripts shows sub-optimal performance. To
address this problem, we propose a simple framework TransliCo that contains
Transliteration Contrastive Modeling (TCM) to fine-tune an mPLM by contrasting
sentences in its training data and their transliterations in a unified script
(Latn, in our case), which ensures uniformity in the representation space for
different scripts. Using Glot500-m, an mPLM pretrained on over 500 languages,
as our source model, we find-tune it on a small portion (5\%) of its training
data, and refer to the resulting model as Furina. We show that Furina not only
better aligns representations from distinct scripts but also outperforms the
original Glot500-m on various crosslingual transfer tasks. Additionally, we
achieve consistent improvement in a case study on the Indic group where the
languages are highly related but use different scripts. We make our code and
models publicly available.

通过 TransliCo 框架中的 Transliteration Contrastive Modeling (TCM) 对 mPLM 进行精调，通过对其训练数据中的句子及其在统一脚本 (Latn) 中的音译进行对比，确保了不同脚本的统一表示空间，解决了多语言预训练语言模型在跨语言知识学习中面临的脚本障碍，并表现出更好的性能。