Artificial Intelligence (AI) has increasingly influenced modern society, recently in particular through significant advancements in Large Language Models (LLMs). However, high computational and storage demands of LLMs still limit their deployment in resource-constrained environments. Knowledge distillation addresses this challenge by training a small student model from a larger teacher model. Previous research has introduced several distillation methods for both generating training data and for training the student model. Despite their relevance, the effects of state-of-the-art distillation methods on model performance and explainability have not been thoroughly investigated and compared. In this work, we enlarge the set of available methods by applying critique-revision prompting to distillation for data generation and by synthesizing existing methods for training. For these methods, we provide a systematic comparison based on the widely used Commonsense Question-Answering (CQA) dataset. While we measure performance via student model accuracy, we employ a human-grounded study to evaluate explainability. We contribute new distillation methods and their comparison in terms of both performance and explainability. This should further advance the distillation of small language models and, thus, contribute to broader applicability and faster diffusion of LLM technology.

本研究解决了大型语言模型在资源受限环境中应用的挑战，通过知识蒸馏训练小型学生模型。我们提出新的蒸馏方法并进行系统比较，发现这些方法在模型性能和可解释性上均有显著提升，推动了小型语言模型的蒸馏进程，为大规模语言模型技术的更广泛应用奠定基础。

宝贝，我缩小了语言模型：知识蒸馏方法对性能和可解释性的影响