The zero-shot cross-lingual ability of models pretrained on multilingual and
even monolingual corpora has spurred many hypotheses to explain this intriguing
empirical result. However, due to the costs of pretraining, most research uses
public models whose pretraining methodology, such as the choice of
tokenization, corpus size, and computational budget, might differ drastically.
When researchers pretrain their own models, they often do so under a
constrained budget, and the resulting models might underperform significantly
compared to SOTA models. These experimental differences led to various
inconsistent conclusions about the nature of the cross-lingual ability of these
models. To help further research on the topic, we released 10 monolingual
byte-level models rigorously pretrained under the same configuration with a
large compute budget (equivalent to 420 days on a V100) and corpora that are 4
times larger than the original BERT's. Because they are tokenizer-free, the
problem of unseen token embeddings is eliminated, thus allowing researchers to
try a wider range of cross-lingual experiments in languages with different
scripts. Additionally, we release two models pretrained on non-natural language
texts that can be used in sanity-check experiments. Experiments on QA and NLI
tasks show that our monolingual models achieve competitive performance to the
multilingual one, and hence can be served to strengthen our understanding of
cross-lingual transferability in language models.

本研究发布了 10 个在相同配置下严格预训练的单语模型，它们是基于字节的，不需要分词，可以消除未见标记嵌入的问题，并且可以在不同文字脚本的语言中尝试更广泛的跨语言实验，通过 QA 和 NLI 任务的实验证明，这些单语模型的性能与多语言模型相当，可以加强我们对语言模型跨语言可迁移性的理解。