Chain-of-thought prompting~(CoT) and tool augmentation have been validated in
recent work as effective practices for improving large language models~(LLMs)
to perform step-by-step reasoning on complex math-related tasks. However, most
existing math reasoning datasets may be not able to fully evaluate and analyze
the ability of LLMs in manipulating tools and performing reasoning, as they may
only require very few invocations of tools or miss annotations for evaluating
intermediate reasoning steps. To address the issue, we construct \textbf{CARP},
a new Chinese dataset consisting of 4,886 computation-intensive algebra
problems with formulated annotations on intermediate steps. In CARP, we test
four LLMs with CoT prompting, and find that they are all prone to make mistakes
at the early steps of the solution, leading to wrong answers. Based on this
finding, we propose a new approach that can deliberate the reasoning steps with
tool interfaces, namely \textbf{DELI}. In DELI, we first initialize a
step-by-step solution based on retrieved exemplars, then iterate two
deliberation procedures that check and refine the intermediate steps of the
generated solution, from the perspectives of tool manipulation and natural
language reasoning, until obtaining converged solutions or reaching the maximum
turn. Experimental results on CARP and six other datasets show that the
proposed DELI mostly outperforms competitive baselines, and can further boost
the performance of existing CoT methods. Our data and code are available in
https://github.com/RUCAIBox/CARP.

该研究构建了一个新的中文数据集 CARP，测试了四个有思考提示的 LLMs，发现它们容易在解决方案的早期步骤中犯错误，因此提出了一个新的基于工具接口的方法 DELI，该方法在 CARP 和其他六个数据集上的实验结果表明 DELI 大多优于竞争基线，能够进一步提高现有的 CoT 方法的性能。