As the capabilities of language models continue to advance, it is conceivable
that "one-size-fits-all" model will remain as the main paradigm. For instance,
given the vast number of languages worldwide, many of which are low-resource,
the prevalent practice is to pretrain a single model on multiple languages. In
this paper, we add to the growing body of evidence that challenges this
practice, demonstrating that monolingual pretraining on the target language
significantly improves models already extensively trained on diverse corpora.
More specifically, we further pretrain GPT-J and LLaMA models on Portuguese
texts using 3% or less of their original pretraining budget. Few-shot
evaluations on Poeta, a suite of 14 Portuguese datasets, reveal that our models
outperform English-centric and multilingual counterparts by a significant
margin. Our best model, Sabi\'a-65B, performs on par with GPT-3.5-turbo. By
evaluating on datasets originally conceived in the target language as well as
translated ones, we study the contributions of language-specific pretraining in
terms of 1) capturing linguistic nuances and structures inherent to the target
language, and 2) enriching the model's knowledge about a domain or culture. Our
results indicate that the majority of the benefits stem from the
domain-specific knowledge acquired through monolingual pretraining.

在这篇论文中，我们证明了在目标语言上进行单语言预训练可以显著提高已经广泛训练于多个语料库的模型，并在 14 个葡萄牙语数据集上表现优于基于英语和多语言模型的模型。我们的结果表明，从单语预训练获得的大多数收益来自于领域特定知识。