Cross-lingual transfer has become an effective way of transferring knowledge
between languages. In this paper, we explore an often-overlooked aspect in this
domain: the influence of the source language of the base language model on
transfer performance. We conduct a series of experiments to determine the
effect of the script and tokenizer used in the pre-trained model on the
performance of the downstream task. Our findings reveal the importance of the
tokenizer as a stronger factor than the sharing of the script, the language
typology match, and the model size.

探索预训练模型的源语言对跨语言转移性能的影响，发现与脚本共享、语言类型匹配和模型大小相比，分词器是更重要的因素。

未知文字：脚本对跨语言迁移的影响

Unknown Script: Impact of Script on Cross-Lingual Transfer

We study the effectiveness of a simple approach to develop a small base
language model (LM) starting from an existing large base LM: first inherit a
few transformer blocks from the larger LM, and then train this smaller model on
a very small subset (0.1\%) of the raw pretraining data of the larger model. We
call our simple recipe Inheritune and first demonstrate it for building a small
base LM with 1.5B parameters using 1B tokens (and a starting few layers of
larger LM of 3B parameters); we do this using a single A6000 GPU for less than
half a day. Across 9 diverse evaluation datasets as well as the MMLU benchmark,
the resulting model compares favorably to publicly available base models of
1B-2B size, some of which have been trained using 50-1000 times more tokens.
We investigate Inheritune in a slightly different setting where we train
small LMs utilizing larger LMs and their full pre-training dataset. Here we
show that smaller LMs trained utilizing some of the layers of GPT2-medium
(355M) and GPT-2-large (770M) can effectively match the val loss of their
bigger counterparts when trained from scratch for the same number of training
steps on OpenWebText dataset with 9B tokens. We analyze our recipe with
extensive experiments and demonstrate it efficacy on diverse settings. Our code
is available at this https URL

我们研究了使用继承和训练的简单方法来构建小型基础语言模型的有效性，并通过实验证明其在各种设置中的功效。