This study proposes a method for distilling the knowledge of fine-tuned Large Language Models (LLMs) into a smaller, more efficient, and accurate neural network, specifically targeting the challenge of deploying these models on resource-constrained devices. Our methodology involves training the smaller student model using the prediction probabilities of the LLM, which serves as a teacher model. This is achieved through a specialized loss function tailored to learn from the LLM's output probabilities, ensuring that the student model closely mimics the teacher's performance. To test this approach, we utilized a large dataset, 7T, containing 6,684 student-written responses to science questions and three other datasets with student-written responses. We also compared performance with original neural network (NN) models to validate the accuracy. Results have shown that the NN and distilled student models have comparable accuracy to the teacher model for the 7T dataset; however, other datasets have shown significantly lower accuracy (28% on average) for NN, though our proposed distilled model is still able to achieve 12\% higher accuracy than NN. Furthermore, the student model size ranges from 0.1M to 0.02M, 100 times smaller in terms of parameters and ten times smaller compared with the original output model size. The significance of this research lies in its potential to make advanced AI technologies accessible in typical educational settings, particularly for automatic scoring.

本研究提出了一种方法，将大型语言模型（LLMs）的知识提炼为一个更小、更高效且准确的神经网络，以实现在资源受限设备上部署这些模型的挑战。我们的方法包括使用LLM的预测概率训练较小的学生模型，作为教师模型，通过专门设计的损失函数来学习LLM的输出概率，确保学生模型能够准确模仿教师模型的性能。通过对包括6,684个学生撰写的科学问题回答及其他数据集的测试，我们将性能与原始神经网络（NN）模型进行了比较，结果显示对于7T数据集，NN模型和提炼的学生模型的准确率与教师模型相当；然而，其他数据集显示NN模型的准确率显著较低（平均28%），然而我们的提炼模型仍然能够比NN模型获得更高12%的准确率。此外，学生模型的参数大小为0.1M至0.02M，相较于原始输出模型大小减小了100倍和10倍。该研究的重要性在于其为自动评分在典型教育环境中的运用提供了潜力。

LLM教育中的知识蒸馏