This study presents a novel learning approach designed to enhance both
mathematical reasoning and problem-solving abilities of Large Language Models
(LLMs). We focus on integrating the Chain-of-Thought (CoT) and the
Program-of-Thought (PoT) learning, hypothesizing that prioritizing the learning
of mathematical reasoning ability is helpful for the amplification of
problem-solving ability. Thus, the initial learning with CoT is essential for
solving challenging mathematical problems. To this end, we propose a sequential
learning approach, named SAAS (Solving Ability Amplification Strategy), which
strategically transitions from CoT learning to PoT learning. Our empirical
study, involving an extensive performance comparison using several benchmarks,
demonstrates that our SAAS achieves state-of-the-art (SOTA) performance. The
results underscore the effectiveness of our sequential learning approach,
marking a significant advancement in the field of mathematical reasoning in
LLMs.

本研究提出了一种新颖的学习方法，旨在增强大型语言模型的数学推理和问题解决能力。通过整合思维链和思维方案的学习，以解决数学问题为重点，提出了一种名为 SAAS 的顺序学习方法。经过广泛的性能比较实证研究，结果表明我们的 SAAS 在数学推理领域取得了最先进的性能。

SAAS：增强大型语言模型数学推理的解决能力放大策略

SAAS: Solving Ability Amplification Strategy for Enhanced Mathematical  Reasoning in Large Language Models

Small-scale models offer various computational advantages, and yet to which
extent size is critical for problem-solving abilities remains an open question.
Specifically for solving grade school math, the smallest model size so far
required to break the 80\% barrier on the GSM8K benchmark remains to be 34B.
Our work studies how high-quality datasets may be the key for small language
models to acquire mathematical reasoning. We introduce \texttt{TinyGSM}, a
synthetic dataset of 12.3M grade school math problems paired with Python
solutions, generated fully by GPT-3.5. After finetuning on \texttt{TinyGSM}, we
find that a duo of a 1.3B generation model and a 1.3B verifier model can
achieve 81.5\% accuracy, outperforming existing models that are orders of
magnitude larger. This also rivals the performance of the GPT-3.5 ``teacher''
model (77.4\%), from which our model's training data is generated. Our approach
is simple and has two key components: 1) the high-quality dataset
\texttt{TinyGSM}, 2) the use of a verifier, which selects the final outputs
from multiple candidate generations.

通过使用小规模模型并结合高质量的数据集，我们的研究探索了如何提升小型语言模型在解决小学数学问题时的数学推理能力，并通过引入 TinyGSM 数据集和验证器的方法，实现了在 GSM8K 基准测试中超过 80% 的准确率，同时在模型规模上远远超过现有模型。

TinyGSM: 使用小型语言模型在 GSM8k 上达到 80% 以上

TinyGSM: achieving >80% on GSM8k with small language models

We explore the creative problem-solving capabilities of modern large language
models (LLMs) in a constrained setting. The setting requires circumventing a
cognitive bias known in psychology as ''functional fixedness'' to use familiar
objects in innovative or unconventional ways. To this end, we create MacGyver,
an automatically generated dataset consisting of 1,600 real-world problems that
deliberately trigger functional fixedness and require thinking
'out-of-the-box'. We then present our collection of problems to both LLMs and
humans to compare and contrast their problem-solving abilities. We show that
MacGyver is challenging for both groups, but in unique and complementary ways.
For example, humans typically excel in solving problems that they are familiar
with but may struggle with tasks requiring domain-specific knowledge, leading
to a higher variance. On the other hand, LLMs, being exposed to a variety of
highly specialized knowledge, attempt broader problems but are prone to
overconfidence and propose actions that are physically infeasible or
inefficient. We also provide a detailed error analysis of LLMs, and demonstrate
the potential of enhancing their problem-solving ability with novel prompting
techniques such as iterative step-wise reflection and divergent-convergent
thinking. This work provides insight into the creative problem-solving
capabilities of humans and AI and illustrates how psychological paradigms can
be extended into large-scale tasks for comparing humans and machines.

我们研究了现代大型语言模型（LLMs）在一个受限制的环境中的创造性问题解决能力。为了达到这个目的，我们创建了一个名为 'MacGyver' 的自动生成数据集，包含 1,600 个故意触发功能固定性并需要 ' 跳出框架 ' 思考的现实世界问题。我们将问题集呈现给 LLMs 和人类，并比较两者的问题解决能力。我们展示了对于两组来说，MacGyver 都是具有挑战性的，但方式是独特且互补的。我们还提供了 LLMs 的详细错误分析，并展示了通过新颖的提示技术（如迭代的逐步反思和发散 - 收敛思维）来增强它们的问题解决能力的潜力。这项工作揭示了人类和人工智能的创造性问题解决能力，并说明了如何将心理学范式扩展到比较人类和机器的大规模任务中。