Closed-source language models such as GPT-4 have achieved remarkable
performance. Many recent studies focus on enhancing the capabilities of smaller
models through knowledge distillation from closed-source language models.
However, due to the incapability to directly access the weights, hidden states,
and output distributions of these closed-source models, the distillation can
only be performed by fine-tuning smaller models with data samples generated by
closed-source language models, which constrains the effectiveness of knowledge
distillation. In this paper, we propose to estimate the output distributions of
closed-source language models within a Bayesian estimation framework, involving
both prior and posterior estimation. The prior estimation aims to derive a
prior distribution by utilizing the corpus generated by closed-source language
models, while the posterior estimation employs a proxy model to update the
prior distribution and derive a posterior distribution. By leveraging the
estimated output distribution of closed-source language models, traditional
knowledge distillation can be executed. Experimental results demonstrate that
our method surpasses the performance of current models directly fine-tuned on
data generated by closed-source language models.

在这篇论文中，我们提出了一种基于贝叶斯估计框架的方法，通过估计闭源语言模型的输出分布来执行传统的知识蒸馏，实验结果表明我们的方法超过了直接在闭源语言模型生成数据上微调的当前模型的性能。