Knowledge distillation from LLMs is essential for the efficient deployment of
language models. Prior works have proposed data generation using LLMs for
preparing distilled models. We argue that generating data with LLMs is prone to
sampling mainly from the center of original content distribution. This
limitation hinders the distilled model from learning the true underlying data
distribution and to forget the tails of the distributions (samples with lower
probability). To this end, we propose GOLD, a task-agnostic data generation and
knowledge distillation framework, which employs an iterative
out-of-distribution-guided feedback mechanism for the LLM. As a result, the
generated data improves the generalizability of distilled models. An
energy-based OOD evaluation approach is also introduced to deal with noisy
generated data. Our extensive experiments on 10 different classification and
sequence-to-sequence tasks in NLP show that GOLD respectively outperforms prior
arts and the LLM with an average improvement of 5% and 14%. We will also show
that the proposed method is applicable to less explored and novel tasks. The
code is available.

基于语言模型的知识蒸馏是提高语言模型的有效部署的关键。本文提出了 GOLD，一种任务无关的数据生成和知识蒸馏框架，通过迭代的超出分布引导的反馈机制来改善生成数据的泛化能力。我们在 10 个不同的分类和序列到序列任务的自然语言处理方面的广泛实验中验证了 GOLD 的优越性和对 less explored 和 novel 任务的适用性。