We study semi-supervised sequence prediction tasks where labeled data are too
scarce to effectively finetune a model and at the same time few-shot prompting
of a large language model (LLM) has suboptimal performance. This happens when a
task, such as parsing, is expensive to annotate and also unfamiliar to a
pretrained LLM. In this paper, we present a discovery that student models
distilled from a prompted LLM can often generalize better than their teacher on
such tasks. Leveraging this finding, we propose a new distillation method,
multistage collaborative knowledge distillation from an LLM (MCKD), for such
tasks. MCKD first prompts an LLM using few-shot in-context learning to produce
pseudolabels for unlabeled data. Then, at each stage of distillation, a pair of
students are trained on disjoint partitions of the pseudolabeled data. Each
student subsequently produces new and improved pseudolabels for the unseen
partition to supervise the next round of student(s) with. We show the benefit
of multistage cross-partition labeling on two constituency parsing tasks. On
CRAFT biomedical parsing, 3-stage MCKD with 50 labeled examples matches the
performance of supervised finetuning with 500 examples and outperforms the
prompted LLM and vanilla KD by 7.5% and 3.7% parsing F1, respectively.

我们提出了一种多阶段协同知识蒸馏方法，用于稀缺标记数据的半监督序列预测任务，通过从提示的大型语言模型中蒸馏出的学生模型，在特定任务上能够更好地泛化，且在两个句法分析任务上表现出了优势。