Knowledge distillation allows smaller neural networks to emulate the performance of larger, teacher models with reduced computational demands. Traditional methods for Large Language Models (LLMs) often necessitate extensive fine-tuning, which limits their accessibility. To address this, we introduce Trace-of-Thought Prompting, a novel framework designed to distill critical reasoning capabilities from high-resource teacher models (over 8 billion parameters) to low-resource student models (up to 8 billion parameters). This approach leverages problem decomposition to enhance interpretability and facilitate human-in-the-loop interventions. Empirical evaluations on the GSM8K and MATH datasets show that student models achieve accuracy gains of up to 113% on GSM8K and 21% on MATH, with significant improvements particularly notable in smaller models like Llama 2 and Zephyr. Our results suggest a promising pathway for open-source, low-resource models to eventually serve both as both students and teachers, potentially reducing our reliance on high-resource, proprietary models.

本研究解决了小型神经网络在知识蒸馏过程中需要大量微调的问题和挑战。我们提出了思维轨迹提示这一新框架，通过问题分解方法，从高资源教学模型中提取关键推理能力，从而提高低资源学生模型的性能。实验证明，学生模型在GSM8K和MATH数据集上具有显著的性能提升，显示出对开放源代码低资源模型的有希望的前景。

思维轨迹提示：通过问题分解研究基于提示的知识蒸馏