Open Japanese large language models (LLMs) have been trained on the Japanese
portions of corpora such as CC-100, mC4, and OSCAR. However, these corpora were
not created for the quality of Japanese texts. This study builds a large
Japanese web corpus by extracting and refining text from the Common Crawl
archive (21 snapshots of approximately 63.4 billion pages crawled between 2020
and 2023). This corpus consists of approximately 312.1 billion characters
(approximately 173 million pages), which is the largest of all available
training corpora for Japanese LLMs, surpassing CC-100 (approximately 25.8
billion characters), mC4 (approximately 239.7 billion characters) and OSCAR
23.10 (approximately 74 billion characters). To confirm the quality of the
corpus, we performed continual pre-training on Llama 2 7B, 13B, 70B, Mistral 7B
v0.1, and Mixtral 8x7B Instruct as base LLMs and gained consistent (6.6-8.1
points) improvements on Japanese benchmark datasets. We also demonstrate that
the improvement on Llama 2 13B brought from the presented corpus was the
largest among those from other existing corpora.

通过从 Common Crawl 档案中提取和精炼文本，该研究构建了一个大型的日语网页语料库，用于训练日语大型语言模型。该语料库包含大约 3121 亿个字符（约 1.73 亿页），是目前可用的日语训练语料库中最大的，超过了 CC-100、mC4 和 OSCAR23.10。通过对基于 Llama 2 的不断预训练，并在日语基准数据集上取得一致（6.6-8.1 分）的改善，该研究证明了所提供语料库对 Llama 2 的改善效果是已有语料库中最大的。