This study presents a novel learning approach designed to enhance both
mathematical reasoning and problem-solving abilities of Large Language Models
(LLMs). We focus on integrating the Chain-of-Thought (CoT) and the
Program-of-Thought (PoT) learning, hypothesizing that prioritizing the learning
of mathematical reasoning ability is helpful for the amplification of
problem-solving ability. Thus, the initial learning with CoT is essential for
solving challenging mathematical problems. To this end, we propose a sequential
learning approach, named SAAS (Solving Ability Amplification Strategy), which
strategically transitions from CoT learning to PoT learning. Our empirical
study, involving an extensive performance comparison using several benchmarks,
demonstrates that our SAAS achieves state-of-the-art (SOTA) performance. The
results underscore the effectiveness of our sequential learning approach,
marking a significant advancement in the field of mathematical reasoning in
LLMs.

本研究提出了一种新颖的学习方法，旨在增强大型语言模型的数学推理和问题解决能力。通过整合思维链和思维方案的学习，以解决数学问题为重点，提出了一种名为 SAAS 的顺序学习方法。经过广泛的性能比较实证研究，结果表明我们的 SAAS 在数学推理领域取得了最先进的性能。

SAAS：增强大型语言模型数学推理的解决能力放大策略

SAAS: Solving Ability Amplification Strategy for Enhanced Mathematical  Reasoning in Large Language Models

Despite the remarkable performance of large language models (LLMs) in recent
NLP tasks, their deployment poses substantial challenges due to high
computational and memory demands. Recent research has concentrated on improving
open-source smaller models through knowledge distillation from LLMs to reduce
computational resource costs with promising outcomes. Nevertheless, they
frequently fall short of attaining LLM-level performance, particularly in tasks
demanding advanced reasoning. In this work, we introduce the \textbf{Mixed
Distillation} framework, which capitalizes on the strengths of
Program-of-Thought (PoT) and Chain-of-Thought (CoT) capabilities within LLMs
and distills these capabilities to smaller models. Regarding these two
capabilities, the PoT is dedicated to enhancing the performance of reasoning
results generated by smaller models, while CoT simultaneously optimizes the
results. Our Mixed Distillation framework offers a promising approach to
enhance the capabilities of smaller models, bridging the gap with LLMs, and
demonstrating better performance across various tasks. Specifically, on the
SVAMP dataset, employing a 7 billion parameter Llama2 and CodeLlama in a mixed
distillation framework not only boosts distillation capabilities beyond
single-path distillation methods but also outperforms the LLM (GPT-3.5-turbo)
in terms of reasoning accuracy. Through sampling in multiple-path reasoning,
the models achieve impressive accuracy performances of 85% and 85.5%,
respectively, signifying advancements over previous distillation methods.

通过 Mixed Distillation 框架，将大语言模型的 Program-of-Thought 和 Chain-of-Thought 的能力转移到较小模型中，提高较小模型的性能，优于传统蒸馏方法，并在多路径推理中实现了令人印象深刻的准确度表现。