With the development of large language models (LLMs), zero-shot learning has
attracted much attention for various NLP tasks. Different from prior works that
generate training data with billion-scale natural language generation (NLG)
models, we propose a retrieval-enhanced framework to create training data from
a general-domain unlabeled corpus. To realize this, we first conduct
contrastive pretraining to learn an unsupervised dense retriever for extracting
the most relevant documents using class-descriptive verbalizers. We then
further propose two simple strategies, namely Verbalizer Augmentation with
Demonstrations and Self-consistency Guided Filtering to improve the topic
coverage of the dataset while removing noisy examples. Experiments on nine
datasets demonstrate that REGEN achieves 4.3% gain over the strongest baselines
and saves around 70% of the time compared to baselines using large NLG models.
Besides, REGEN can be naturally integrated with recently proposed large
language models to boost performance.

本研究提出了一个检索增强的框架，使用特征描述性词语作为 dense retriever，从无标注的语料库中创建训练数据，以便在与基于大型自然语言生成模型相比节省 70% 的时间内，取得优于最强基线 4.3% 的成绩。