Large language models (LLMs) have demonstrated remarkable open-domain
capabilities. Traditionally, LLMs tailored for a domain are trained from
scratch to excel at handling domain-specific tasks. In this work, we explore an
alternative strategy of continual pre-training as a means to develop
domain-specific LLMs. We introduce FinPythia-6.9B, developed through
domain-adaptive continual pre-training on the financial domain. Continual
pre-trained FinPythia showcases consistent improvements on financial tasks over
the original foundational model. We further explore simple but effective data
selection strategies for continual pre-training. Our data selection strategies
outperforms vanilla continual pre-training's performance with just 10% of
corpus size and cost, without any degradation on open-domain standard tasks.
Our work proposes an alternative solution to building domain-specific LLMs from
scratch in a cost-effective manner.

本研究探讨了持续预训练作为开发面向特定领域的大型语言模型的一种替代策略，引入了通过领域自适应持续预训练在金融领域上开发的 FinPythia-6.9B，FinPythia 通过金融任务中持续预训练实现了稳定的改进，并进一步探索了简单而有效的数据选择策略，相较于常规的持续预训练方法，我们的数据选择策略在使用仅 10% 的语料库大小和成本时表现更好，在开放领域的标准任务上没有任何退化。本研究提出了一种以成本效益的方式构建面向特定领域的大型语言模型的替代解决方案。

构建领域特定的大型语言模型的高效连续预训练

Efficient Continual Pre-training for Building Domain Specific Large  Language Models

Few-shot transfer often shows substantial gain over zero-shot
transfer~\cite{lauscher2020zero}, which is a practically useful trade-off
between fully supervised and unsupervised learning approaches for multilingual
pretrained model-based systems. This paper explores various strategies for
selecting data for annotation that can result in a better few-shot transfer.
The proposed approaches rely on multiple measures such as data entropy using
$n$-gram language model, predictive entropy, and gradient embedding. We propose
a loss embedding method for sequence labeling tasks, which induces diversity
and uncertainty sampling similar to gradient embedding. The proposed data
selection strategies are evaluated and compared for POS tagging, NER, and NLI
tasks for up to 20 languages. Our experiments show that the gradient and loss
embedding-based strategies consistently outperform random data selection
baselines, with gains varying with the initial performance of the zero-shot
transfer. Furthermore, the proposed method shows similar trends in improvement
even when the model is fine-tuned using a lower proportion of the original
task-specific labeled training data for zero-shot transfer.

本文探索了各种数据选择策略，依靠多个度量，利用了渐变嵌入和损失嵌入方法，以提高针对多语言预训练模型的 POS 标记、NER 和 NLI 任务的 Few-shot 迁移的性能。实验证明，所提出的方法始终优于随机数据选择基线，即使仅使用更少的标记数据进行零 - shot 迁移，也能显著提高性能。