Large multilingual pretrained language models (mPLMs) have become the de
facto state of the art for cross-lingual transfer in NLP. However, their
large-scale deployment to many languages, besides pretraining data scarcity, is
also hindered by the increase in vocabulary size and limitations in their
parameter budget. In order to boost the capacity of mPLMs to deal with
low-resource and unseen languages, we explore the potential of leveraging
transliteration on a massive scale. In particular, we explore the UROMAN
transliteration tool, which provides mappings from UTF-8 to Latin characters
for all the writing systems, enabling inexpensive romanization for virtually
any language. We first focus on establishing how UROMAN compares against other
language-specific and manually curated transliterators for adapting
multilingual PLMs. We then study and compare a plethora of data- and
parameter-efficient strategies for adapting the mPLMs to romanized and
non-romanized corpora of 14 diverse low-resource languages. Our results reveal
that UROMAN-based transliteration can offer strong performance for many
languages, with particular gains achieved in the most challenging setups: on
languages with unseen scripts and with limited training data without any
vocabulary augmentation. Further analyses reveal that an improved tokenizer
based on romanized data can even outperform non-transliteration-based methods
in the majority of languages.

本文研究了利用大量转写大幅提高多语种预训练语言模型在少资源语言中的性能，并且发现使用 UROMAN 基于的转写方法可以在许多语言中提供强大的性能，特别是在对未见到的语言脚本和数据量有限的情况下。