Cross-lingual continual pre-training of large language models (LLMs)
initially trained on English corpus allows us to leverage the vast amount of
English language resources and reduce the pre-training cost. In this study, we
constructed Swallow, an LLM with enhanced Japanese capability, by extending the
vocabulary of Llama 2 to include Japanese characters and conducting continual
pre-training on a large Japanese web corpus. Experimental results confirmed
that the performance on Japanese tasks drastically improved through continual
pre-training, and the performance monotonically increased with the amount of
training data up to 100B tokens. Consequently, Swallow achieved superior
performance compared to other LLMs that were trained from scratch in English
and Japanese. An analysis of the effects of continual pre-training revealed
that it was particularly effective for Japanese question answering tasks.
Furthermore, to elucidate effective methodologies for cross-lingual continual
pre-training from English to Japanese, we investigated the impact of vocabulary
expansion and the effectiveness of incorporating parallel corpora. The results
showed that the efficiency gained through vocabulary expansion had no negative
impact on performance, except for the summarization task, and that the combined
use of parallel corpora enhanced translation ability.

本研究通过跨语言连续预训练来提升大规模语言模型的日语能力，并扩展了词汇表以包括日语字符，通过在大规模网页语料库上进行连续预训练，结果表明，Swallow 在日语任务上的性能显著提高，并且随着训练数据量的增加而逐渐增强，相比从头开始训练的其他英日语言模型，Swallow 表现出较优异的性能。同时，研究还探究了从英语到日语的跨语言连续预训练的有效方法，发现词汇扩展对性能的效果除了摘要任务外没有负面影响，且使用平行语料库可以提高翻译能力。