Current state-of-the-art dialogue systems heavily rely on extensive training
datasets. However, challenges arise in domains where domain-specific training
datasets are insufficient or entirely absent. To tackle this challenge, we
propose a novel data \textbf{A}ugmentation framework for
\textbf{M}ulti-\textbf{D}omain \textbf{D}ialogue \textbf{G}eneration, referred
to as \textbf{AMD$^2$G}. The AMD$^2$G framework consists of a data augmentation
process and a two-stage training approach: domain-agnostic training and domain
adaptation training. We posit that domain corpora are a blend of
domain-agnostic and domain-specific features, with certain representation
patterns shared among diverse domains. Domain-agnostic training aims to enable
models to learn these common expressive patterns. To construct domain-agnostic
dialogue corpora, we employ a \textit{\textbf{de-domaining}} data processing
technique used to remove domain-specific features. By mitigating the effects of
domain-specific features, the model trained on the de-domained corpora can
effectively learn common expression patterns in different domains.
Subsequently, we adapt the learned domain-agnostic features to the target
domain through domain adaptation training. We conduct experiments on Chinese
dialogue datasets from five different domains and show that AMD$^2$G achieves
superior performance compared to both direct training on the target domain
corpus and collective training on all five domain corpora. Our work underscores
AMD$^2$G as a viable alternative solution for low-resource multi-domain
dialogue generation. Code and data associated with our work are available on
GitHub repository$^{\text 1}$.

我们提出了一种用于多领域对话生成的新型数据增强框架 AMD2G，它由数据增强过程和两阶段训练方法组成，域无关训练旨在使模型学习共同的表达模式，然后通过域自适应训练来调整学到的域无关特征到目标域。实验证明，与直接在目标域语料上训练和在所有五个域上进行集合训练相比，AMD2G 能够在五个不同领域的中文对话数据集上取得更优异的性能。

低资源多领域对话生成的统一数据增强框架

A Unified Data Augmentation Framework for Low-Resource Multi-Domain  Dialogue Generation

Pretrained Large Language Models (LLM) such as ChatGPT, Claude, etc. have
demonstrated strong capabilities in various fields of natural language
generation. However, there are still many problems when using LLM in
specialized domain-specific fields. When using generative AI to process
downstream tasks, a common approach is to add new knowledge (e.g., private
domain knowledge, cutting-edge information) to a pretrained model through
continued training or fine-tuning. However, whether there is a universal
paradigm for domain adaptation training is still an open question. In this
article, we proposed Information Gain Optimized Tokenizer (IGOT), which
analyzes the special token set of downstream tasks, constructs a new subset
using heuristic function $\phi$ with the special token and its information
gain, to build new domain-specific tokenizer, and continues pretraining on the
downstream task data. We explored the many positive effects of this method's
customized tokenizer on domain-adaptive pretraining and verified this method
can perform better than the ordinary method of just collecting data and
fine-tuning. Based on our experiment, the continued pretraining process of IGOT
with LLaMA-7B achieved 11.9\% token saving, 12.2\% training time saving, and
5.8\% maximum GPU VRAM usage saving, combined with the T5 model, we can even
reach a 31.5\% of training time saving, making porting general generative AI to
specific domains more effective than before. In domain-specific tasks,
supervised $IGOT_\tau$ shows great performance on reducing both the convergence
radius and convergence point during keep pretraining.

使用信息增益优化的分词器 (IGOT) 在特定领域下进行持续预训练可提高训练效率与性能，降低收敛半径与收敛点。