Several previous studies have considered language- and domain-specific large
language models (LLMs) as separate topics. This study explores the combination
of a non-English language and a high-demand industry domain, focusing on a
Japanese business-specific LLM. This type of a model requires expertise in the
business domain, strong language skills, and regular updates of its knowledge.
We trained a 13-billion-parameter LLM from scratch using a new dataset of
business texts and patents, and continually pretrained it with the latest
business documents. Further we propose a new benchmark for Japanese business
domain question answering (QA) and evaluate our models on it. The results show
that our pretrained model improves QA accuracy without losing general
knowledge, and that continual pretraining enhances adaptation to new
information. Our pretrained model and business domain benchmark are publicly
available.

本研究通过结合非英语语言和高需求产业领域的研究，探讨了一种特定于日本商业领域的大型语言模型（LLM）。我们从头开始训练了一个 130 亿参数的 LLM，使用了一组新的商业文本和专利，并持续使用最新的商业文件进行预训练。随后，我们提出了一个新的日本商业领域问题回答（QA）基准，并对我们的模型进行了评估。结果表明，我们的预训练模型提高了 QA 的准确性，并且在不损失一般知识的情况下不断适应新信息。我们的预训练模型和商业领域基准已公开提供。