Despite rapid progress in large language models (LLMs), their performance on
a vast majority of languages remain unsatisfactory. In this paper, we study
building language-specific LLMs by adapting monolingual and multilingual LLMs.
We conduct systematic experiments on how design choices (base model selection,
vocabulary extension, and continued fine-tuning) impact the adapted LLM, both
in terms of efficiency (how many tokens are needed to encode the same amount of
information) and end task performance. We find that (1) the initial performance
before the adaptation is not always indicative of the final performance. (2)
Efficiency can easily improved with simple vocabulary extension and continued
fine-tuning in most LLMs we study, and (3) The optimal adaptation method is
highly language-dependent, and the simplest approach works well across various
experimental settings. Adapting English-centric models can yield better results
than adapting multilingual models despite their worse initial performance on
low-resource languages. Together, our work lays foundations on efficiently
building language-specific LLMs by adapting existing LLMs.

通过对已有大语言模型进行适应和扩展，我们研究了构建语言专属的大语言模型。我们通过系统实验探究基础模型选择、词汇扩展和持续微调等设计选择对适应后的大语言模型的效率（编码同样数量信息所需的词汇数）和最终任务性能的影响。我们发现，（1）适应前的初始性能并不总是最终性能的指示；（2）大多数研究的大语言模型可以通过简单的词汇扩展和持续微调来提高效率；（3）最佳的适应方法高度依赖于语言，简单的方法在各种实验设置中都表现良好。与适应多语言模型相比，适应以英语为中心的模型在资源稀缺语言上可以取得更好的结果。总之，我们的工作为通过适应现有大语言模型高效构建语言专属大语言模型奠定了基础。