Knowledge distillation (KD) has been widely employed to transfer knowledge from a large language model (LLM) to a specialized model in low-data regimes through pseudo label learning. However, pseudo labels generated by teacher models are usually noisy and may influence KD performance. This study delves into KD with noisy teachers and uncovers that the student model can already generate more accurate predictions than the teacher labels used to train it during KD, indicating its inherent ability to denoise noisy teacher labels. Motivated by this finding, we propose Peer-Advised KD to improve vanilla KD from noisy teachers. Experiments show that Peer-Advised KD can outperform LLM by approximately 5% with 50 human-labeled data, and even competitive to standard supervised finetuning with 750 human-labeled data.

通过伪标签学习，将大型语言模型（LLM）的知识在低数据环境中转移给专门的模型，本研究探索了使用含噪声教师模型进行知识蒸馏的情况，并发现学生模型在知识蒸馏过程中能够生成比教师标签更准确的预测结果，表明其具有去噪噪声教师标签的内在能力，基于此发现，我们提出了使用对等建议（Peer-Advised KD）来改进标准知识蒸馏方法，实验证明，使用50个人工标注数据，Peer-Advised KD 的性能优于LLM约5％，甚至与使用750个人工标注数据进行的标准监督微调相媲美。

学生作为噪声教师的内在降噪器