*Data Synthesis* is a promising way to train a small model with very little
labeled data. One approach for data synthesis is to leverage the rich knowledge
from large language models to synthesize pseudo training examples for small
models, making it possible to achieve both data and compute efficiency at the
same time. However, a key challenge in data synthesis is that the synthesized
dataset often suffers from a large distributional discrepancy from the *real
task* data distribution. Thus, in this paper, we propose *Synthesis Step by
Step* (**S3**), a data synthesis framework that shrinks this distribution gap
by iteratively extrapolating the errors made by a small model trained on the
synthesized dataset on a small real-world validation dataset using a large
language model. Extensive experiments on multiple NLP tasks show that our
approach improves the performance of a small model by reducing the gap between
the synthetic dataset and the real data, resulting in significant improvement
compared to several baselines: 9.48% improvement compared to ZeroGen and 2.73%
compared to GoldGen, and at most 15.17% improvement compared to the small model
trained on human-annotated data.

通过使用大型语言模型来进行数据合成，我们提出了一种名为 “Synthesis Step by Step (S3)” 的数据合成框架，该框架通过迭代地使用大型语言模型对小型真实验证数据集上的小型模型合成数据集中的错误进行外推，从而缩小合成数据集与真实数据之间的分布差异。在多个自然语言处理任务的广泛实验中表明，我们的方法能够通过减小合成数据集与真实数据之间的差距来提高小型模型的性能，相较于 ZeroGen 和 GoldGen，我们的方法改进了 9.48% 和 2.73%，并且相较于人工标注数据训练的小型模型最多提高了 15.17%。