Multilingual pre-trained models (mPLMs) have shown impressive performance on
cross-lingual transfer tasks. However, the transfer performance is often
hindered when a low-resource target language is written in a different script
than the high-resource source language, even though the two languages may be
related or share parts of their vocabularies. Inspired by recent work that uses
transliteration to address this problem, our paper proposes a
transliteration-based post-pretraining alignment (PPA) method aiming to improve
the cross-lingual alignment between languages using diverse scripts. We select
two areal language groups, $\textbf{Mediterranean-Amharic-Farsi}$ and
$\textbf{South+East Asian Languages}$, wherein the languages are mutually
influenced but use different scripts. We apply our method to these language
groups and conduct extensive experiments on a spectrum of downstream tasks. The
results show that after PPA, models consistently outperform the original model
(up to 50% for some tasks) in English-centric transfer. In addition, when we
use languages other than English as sources in transfer, our method obtains
even larger improvements. We will make our code and models publicly available
at https://github.com/cisnlp/Transliteration-PPA.

使用基于音译的预训练后对齐（PPA）方法，改善使用不同文字脚本的相互关联低资源目标语言与高资源源语言之间的跨语言对齐，进而在跨语言转移任务中，提高模型性能达到 50% 的改进；同时使用非英语语言作为源语言转移，可获得更大的改进。