Large Language Models (LLMs) have shown superior performance in various
applications and fields. To achieve better performance on specialized domains
such as law and advertisement, LLMs are often continue pre-trained on in-domain
data. However, existing approaches suffer from two major issues. First,
in-domain data are scarce compared with general domain-agnostic data. Second,
data used for continual pre-training are not task-aware, such that they may not
be helpful to downstream applications. We propose TRAIT, a task-oriented
in-domain data augmentation framework. Our framework is divided into two parts:
in-domain data selection and task-oriented synthetic passage generation. The
data selection strategy identifies and selects a large amount of in-domain data
from general corpora, and thus significantly enriches domain knowledge in the
continual pre-training data. The synthetic passages contain guidance on how to
use domain knowledge to answer questions about downstream tasks. By training on
such passages, the model aligns with the need of downstream applications. We
adapt LLMs to two domains: advertisement and math. On average, TRAIT improves
LLM performance by 8% in the advertisement domain and 7.5% in the math domain.

利用任务导向的领域特定数据增强框架，从通用语料库中选择大量领域内数据并生成合成的领域特定文章，以提升大语言模型在广告和数学领域的表现。平均来说，该框架可使广告领域性能提高 8%，数学领域提高 7.5%。