In recent years, large language models (LLMs) have shown exceptional capabilities across various natural language processing (NLP) tasks. However, such impressive performance often comes with the trade-off of an increased parameter size, posing significant challenges for widespread deployment. Knowledge distillation (KD) provides a solution by transferring knowledge from a large teacher model to a smaller student model. In this paper, we explore the task-specific distillation of LLMs at the logit level. Our investigation reveals that the logits of fine-tuned LLMs exhibit a more extreme long-tail distribution than those from vision models, with hidden "noise" in the long tail affecting distillation performance. Furthermore, existing logits distillation methods often struggle to effectively utilize the internal ranking information from the logits. To address these, we propose the Bi-directional Logits Difference (BiLD) loss. The BiLD loss filters out the long-tail noise by utilizing only top-$k$ teacher and student logits, and leverages the internal logits ranking information by constructing logits differences. To evaluate BiLD loss, we conduct comprehensive experiments on 13 datasets using two types of LLMs. Our results show that the BiLD loss, with only the top-8 logits, outperforms supervised fine-tuning (SFT), vanilla KL loss, and five other distillation methods from both NLP and CV fields.

本研究探讨了基于对数几率水平的大型语言模型任务特定蒸馏，发现与视觉模型相比，微调后的大型语言模型的对数几率呈现更极端的长尾分布，长尾部分中的噪声对蒸馏性能产生影响。提出了双向对数几率差异损失（BiLD loss）作为解决方案，并通过构建对数几率之间的差异来利用内部对数几率的排序信息。实验结果表明，仅使用前8个对数几率的BiLD损失优于有监督微调（SFT），基本KL损失和其他NLP和CV领域的五种蒸馏方法。

BiLD: 大型语言模型蒸馏中的双向对数损失差异