Large language models (LLMs) have demonstrated remarkable open-domain
capabilities. Traditionally, LLMs tailored for a domain are trained from
scratch to excel at handling domain-specific tasks. In this work, we explore an
alternative strategy of continual pre-training as a means to develop
domain-specific LLMs. We introduce FinPythia-6.9B, developed through
domain-adaptive continual pre-training on the financial domain. Continual
pre-trained FinPythia showcases consistent improvements on financial tasks over
the original foundational model. We further explore simple but effective data
selection strategies for continual pre-training. Our data selection strategies
outperforms vanilla continual pre-training's performance with just 10% of
corpus size and cost, without any degradation on open-domain standard tasks.
Our work proposes an alternative solution to building domain-specific LLMs from
scratch in a cost-effective manner.

本研究探讨了持续预训练作为开发面向特定领域的大型语言模型的一种替代策略，引入了通过领域自适应持续预训练在金融领域上开发的 FinPythia-6.9B，FinPythia 通过金融任务中持续预训练实现了稳定的改进，并进一步探索了简单而有效的数据选择策略，相较于常规的持续预训练方法，我们的数据选择策略在使用仅 10% 的语料库大小和成本时表现更好，在开放领域的标准任务上没有任何退化。本研究提出了一种以成本效益的方式构建面向特定领域的大型语言模型的替代解决方案。