This work addresses the challenge of democratizing advanced Large Language
Models (LLMs) by compressing their mathematical reasoning capabilities into
sub-billion parameter Small Language Models (SLMs) without compromising
performance. We introduce Equation-of-Thought Distillation (EoTD), a novel
technique that encapsulates the reasoning process into equation-based
representations to construct an EoTD dataset for fine-tuning SLMs.
Additionally, we propose the Mix Thoughts Distillation (MTD) framework to
enhance the reasoning performance of SLMs. This involves creating a reasoning
dataset with multiple thought processes and using it for fine-tuning. Our
experimental findings demonstrate that EoTD significantly boosts the reasoning
abilities of SLMs, while MTD enables these models to achieve state-of-the-art
reasoning performance.

通过将大型语言模型的数学推理能力压缩到亿级参数的小型语言模型中，不影响性能，以实现先进的大型语言模型民主化。我们提出了 EoTD（Equation-of-Thought Distillation）技术，将推理过程封装到基于方程的表示中，构建了 EoTD 数据集用于微调小型语言模型。此外，我们提出了 MTD（Mix Thoughts Distillation）框架，以增强小型语言模型的推理性能。实验结果表明，EoTD 显著提升了小型语言模型的推理能力，而 MTD 使这些模型达到了最先进的推理性能。