Tool-augmented Large Language Models (TALM) are known to enhance the skillset
of large language models (LLM), thereby, leading to their improved reasoning
abilities across many tasks. While, TALMs have been successfully employed in
different question-answering benchmarks, their efficacy on complex mathematical
reasoning benchmarks, and the potential complimentary benefits offered by tools
for knowledge retrieval and mathematical equation solving, are open research
questions. In this work, we present MATHSENSEI, a tool-augmented large language
model for mathematical reasoning. Augmented with tools for knowledge retrieval
(Bing Web Search), program execution (Python), and symbolic equation solving
(Wolfram-Alpha), we study the complimentary benefits of these tools through
evaluations on mathematical reasoning datasets. We perform exhaustive ablations
on MATH,a popular dataset for evaluating mathematical reasoning on diverse
mathematical disciplines. We also conduct experiments involving well-known tool
planners to study the impact of tool sequencing on the model performance.
MATHSENSEI achieves 13.5% better accuracy over gpt-3.5-turbo with
chain-of-thought on the MATH dataset. We further observe that TALMs are not as
effective for simpler math word problems (in GSM-8k), and the benefit increases
as the complexity and required knowledge increases (progressively over AQuA,
MMLU-Math, and higher level complex questions in MATH). The code and data are
available at this https URL

MATHSENSEI 是一个工具增强的大型语言模型，通过添加知识检索、程序执行和符号方程求解的工具，针对数学推理问题进行评估，表现出比 gpt-3.5-turbo 以及基于 chain-of-thought 的 MATH 数据集上准确率提高了 13.5%。同时我们还观察到，TALMs 在简单数学问题上的效果不如在复杂问题上的效果显著，且随着问题复杂程度和所需知识的增加而提高。