Pre-trained multilingual language models such as mBERT and XLM-R have
demonstrated great potential for zero-shot cross-lingual transfer to low
web-resource languages (LRL). However, due to limited model capacity, the large
difference in the sizes of available monolingual corpora between high
web-resource languages (HRL) and LRLs does not provide enough scope of
co-embedding the LRL with the HRL, thereby affecting downstream task
performance of LRLs. In this paper, we argue that relatedness among languages
in a language family along the dimension of lexical overlap may be leveraged to
overcome some of the corpora limitations of LRLs. We propose Overlap BPE
(OBPE), a simple yet effective modification to the BPE vocabulary generation
algorithm which enhances overlap across related languages. Through extensive
experiments on multiple NLP tasks and datasets, we observe that OBPE generates
a vocabulary that increases the representation of LRLs via tokens shared with
HRLs. This results in improved zero-shot transfer from related HRLs to LRLs
without reducing HRL representation and accuracy. Unlike previous studies that
dismissed the importance of token-overlap, we show that in the low-resource
related language setting, token overlap matters. Synthetically reducing the
overlap to zero can cause as much as a four-fold drop in zero-shot transfer
accuracy.

本文提出了一种名为 Overlap BPE 的用于增强低网络资源语言在多语言模型中表示的词汇生成算法，并通过多项实验表明，通过增加相关语言之间的词汇重叠度，可以在不降低高网络资源语言表示和准确性的前提下，提高相关低网络资源语言的零样本学习的效果。