Chain-of-Thought and Program-Aided Language Models represent two distinct
reasoning methods, each with its own strengths and weaknesses. We demonstrate
that it is possible to combine the best of both worlds by using different
models for different problems, employing a large language model (LLM) to
perform model selection. Through a theoretical analysis, we discover that the
performance improvement is determined by the differences between the combined
methods and the success rate of choosing the correct model. On eight reasoning
datasets, our proposed approach shows significant improvements. Furthermore, we
achieve new state-of-the-art results on GSM8K and SVAMP with accuracies of
96.5% and 93.7%, respectively. Our code is publicly available at
this https URL

本研究通过使用不同的模型进行模型选择，在充分分析理论基础上，在八个推理数据集上实现了显著的性能改进，并在 GSM8K 和 SVAMP 上实现了 96.5％和 93.7％的准确率，达到了最新的研究成果。

基于大语言模型的推理自动模型选择

Automatic Model Selection with Large Language Models for Reasoning

Large language models (LLMs) have recently demonstrated an impressive ability
to perform arithmetic and symbolic reasoning tasks, when provided with a few
examples at test time ("few-shot prompting"). Much of this success can be
attributed to prompting methods such as "chain-of-thought'', which employ LLMs
for both understanding the problem description by decomposing it into steps, as
well as solving each step of the problem. While LLMs seem to be adept at this
sort of step-by-step decomposition, LLMs often make logical and arithmetic
mistakes in the solution part, even when the problem is decomposed correctly.
In this paper, we present Program-Aided Language models (PAL): a novel approach
that uses the LLM to read natural language problems and generate programs as
the intermediate reasoning steps, but offloads the solution step to a runtime
such as a Python interpreter. With PAL, decomposing the natural language
problem into runnable steps remains the only learning task for the LLM, while
solving is delegated to the interpreter. We demonstrate this synergy between a
neural LLM and a symbolic interpreter across 13 mathematical, symbolic, and
algorithmic reasoning tasks from BIG-Bench Hard and other benchmarks. In all
these natural language reasoning tasks, generating code using an LLM and
reasoning using a Python interpreter leads to more accurate results than much
larger models. For example, PAL using Codex achieves state-of-the-art few-shot
accuracy on the GSM8K benchmark of math word problems, surpassing PaLM-540B
which uses chain-of-thought by absolute 15% top-1. Our code and data are
publicly available at this http URL .

本论文介绍了一种新颖的方法，使用大型语言模型来读取自然语言问题并生成程序作为中间推理步骤，但将求解步骤委托给运行时，如 Python 解释器，在 13 个数学、符号和算法推理任务中展示了神经大型语言模型和符号解释器之间的协同作用。