This paper focuses on the data augmentation for low-resource NLP tasks where
the training set is limited. The existing solutions either leverage
task-independent heuristic rules (e.g., Synonym Replacement) or fine-tune
general-purpose pre-trained language models (e.g., GPT2) using the limited
training instances to produce new synthetic data. Consequently, they have
trivial task-specific knowledge and are limited to yielding low-quality
synthetic data. To combat this issue, we propose Knowledge Mixture Data
Augmentation Model (KnowDA) which is an Seq2Seq language model pre-trained on a
mixture of diverse NLP tasks under a novel framework of Knowledge Mixture
Training (KoMT). The goal of KoMT is to condense diverse NLP task-specific
knowledge into the single KnowDA model (i.e., all-in-one) such that KnowDA
could utilize these knowledge to quickly grasp the inherent synthesis law of
the target task through limited training instances. Specifically, KoMT
reformulates input examples from various heterogeneous NLP tasks into a unified
text-to-text format, and employs denoising training objectives in different
granularity to learn to reconstruct partial or complete samples. To the best of
our knowledge, we are the first attempt to apply 100+ NLP multi-task training
for data augmentation. Extensive experiments show that i) the synthetic data
produced by KnowDA successfully improves performance of the strong pre-trained
language models (i.e., Bert, ALBert and Deberta) by a large margin on the
low-resource NLP benchmark FewGLUE, CoNLL'03 and WikiAnn; ii) KnowDA
successfully transfers the task knowledge to NLP tasks whose types are seen and
unseen in KoMT.

该论文提出了一种名为 KnowDA 的 Seq2Seq 语言模型，该模型利用知识混合训练 (KoMT) 在多个异构 NLP 任务中预先训练并生成高质量的合成数据，用于解决低资源 NLP 任务中数据增强的问题。实验证明，KnowDA 成功地将任务知识转移给在 KoMT 中看到和没有看到过的 NLP 任务。